Apple introduces DeepAmbigQA dataset to test LLM answer completeness on ambiguous multi-hop questions

Apple’s Machine Learning Research team introduced DeepAmbigQA, a dataset of 3,600 questions designed to evaluate large language models (LLMs) on ambiguous multi-hop reasoning. The dataset is constructed using DEEPAMBIGQAGEN, an automatic data generation pipeline that creates QA tasks grounded in text corpora and linked knowledge graphs. This pipeline generates natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning, two challenges that existing QA benchmarks rarely evaluate jointly.

Half of the questions in DeepAmbigQA explicitly require resolving name ambiguity—such as distinguishing between multiple films sharing the same title—while all questions demand multi-step evidence gathering and integration. For example, the dataset includes complex questions like “Which actor from the film Heat won at least one Academy Award?”, which requires identifying the correct film, retrieving the cast, and then filtering for actors with Academy Awards.

In experiments reported by the authors, even state-of-the-art models such as GPT-5 showed incomplete answers on the dataset. The models achieved an exact match score of 0.13 on ambiguous questions and 0.21 on non-ambiguous questions, indicating significant room for improvement in producing complete and accurate answer sets.

The release includes both the DeepAmbigQA dataset and the DEEPAMBIGQAGEN pipeline, enabling researchers to generate additional tasks and extend the benchmark. The authors emphasize that the findings highlight the need for more robust QA systems capable of handling information gathering and answer completeness, especially in open-domain settings where ambiguity and multi-step reasoning are common.