OpenAI’s math proofs fall short of standards set by external advisory group

OpenAI released hundreds of claimed solutions to difficult mathematical problems this week, asserting that it had consulted an external advisory group of elite mathematicians to avoid past controversies. However, the Advisory Group on Mathematics and Artificial Intelligence (AGMAI), hosted by Princeton University’s Institute for Advanced Study, said OpenAI’s latest proofs did not meet the group’s guidelines, particularly around human understanding and formalization of results.

AGMAI’s first guideline was to ask frontier labs to stop testing advanced mathematical problems on proprietary models. OpenAI’s release explicitly stated it was evaluating proprietary models using open research problems in mathematics, contradicting that recommendation. AGMAI also emphasized that proofs should be formalized and that human mathematicians should be involved in validating AI-generated solutions. OpenAI released results with limited transparency: only 10 of 719 manuscripts included the model’s chain of thought, and 42% of the proofs had not been formalized.

A separate paper by mathematicians at the University of Cambridge and King’s College London examined discrepancies between natural-language proofs and their formalized Lean code in OpenAI’s releases. The authors argue that without human oversight, such translations can introduce errors, and they call for the same peer review scrutiny applied to human proofs. AGMAI had also asked OpenAI to include machine-readable metadata correlating natural-language and formal artifacts, which OpenAI did not do in these releases.

Mathematicians stress that when humans solve new problems, they take responsibility by engaging with the community through papers, talks, and seminars to ensure understanding and broader applicability. Harvard mathematics professor Melanie Wood told TechCrunch that AI-generated solutions often lack human understanding at the point of release, shifting the burden of validation to the community after the fact.