Cheap open-weight models match frontier LLMs in grading math proofs at up to 100x lower cost

A new arXiv preprint evaluates whether low-cost open-weight models can serve as reliable judges for grading natural-language mathematical proofs, a task that typically relies on expensive frontier LLMs. The study tests three budget-friendly models—GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B—on a 200-instance validation subset of the IMO-GradingBench benchmark. The authors report that these models achieve pass/fail grading decisions statistically indistinguishable from human evaluators, matching the performance of frontier judges like Claude Opus 4.7 and Gemini 3.1 Pro.

Extending the evaluation to the full 1,000-instance benchmark, the researchers explore consensus rules for aggregating model judgments. They find that requiring unanimous agreement (all-three-pass) maximizes pass-agreement and precision while minimizing variability across four replicate runs. Notably, this consensus rule outperformed a simple majority vote, which matched frontier performance but did not improve upon the strongest individual model.

The headline result is that cheap judges can match frontier systems at up to 100 times lower cost, with the authors recommending the all-three-pass rule as a deployable default. They caution, however, that this rule was identified post-hoc and requires independent replication before widespread adoption. The study also notes that the benchmark and rubric used—IMO-GradingBench—are designed to reflect human grading standards, providing a realistic testbed for automated judging of mathematical proofs.