Paper proposes label-free method to detect when multi-agent code judges lack grounding

A new paper on arXiv proposes a label-free method to determine when multi-agent code judges lack grounding in their evaluations. The authors argue that current multi-agent verification frameworks, which decompose judgments into checkable claims and verify them against evidence, often fail in code judging scenarios because the evidence used is not independent of the answer under review and does not differ meaningfully between compared solutions.

The researchers evaluated a published framework, MARCH, across 80 condition-by-cell measurements on two code judging benchmarks. They found that MARCH declared both solutions equally good in 78% to 95% of comparisons, achieving only 4.4% accuracy compared to 43.7% when the same model was asked directly. Neither simplifying the problems nor increasing the judge’s capacity resolved this issue.

To address the problem, the authors extracted two measurements from the pipeline’s own logs that indicate when the judge lacks sufficient evidence. By applying one of these measurements as a gate, the pipeline declined to answer comparisons where it could not ground its judgment, improving accuracy from 20.7% to 36.9% while still responding to half of all comparisons. The contribution is framed not as a more accurate judge, but as a way to identify when a judge has no basis for its answer.

The paper is titled 'When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess' and was submitted to the 21st Women in Machine Learning Workshop (WiML) at NeurIPS 2026.