Study finds reinforcement learning repairs LLM falsification gaps in theorem disproof

Researchers propose framing counterexample generation as constrained witness emission against a deterministic per-theorem Python verifier, releasing SymCE, a corpus of 4,707 false undergraduate-algebra and real-analysis conjectures each paired with executable verifiers. The verifier also functions as the reward function, turning SymCE into a training environment for counterexample generation.

In experiments with Qwen3-4B, supervised fine-tuning on counterexamples alone produced an "imitation trap," collapsing true-theorem recognition from 0.27 to 0.00. The collapse replicated across four random seeds and on Gemma-3-4B, indicating robustness of the failure mode.

Reinforcement learning with a sparse outcome-only reward repaired the collapse, restoring true-theorem recognition to 0.66. Sparse and dense rewards achieved statistically indistinguishable in-domain success, but diverged by 33 points on a held-out calibration probe, a gap the authors attribute to the partial-credit term used in dense rewards.

The 4B-parameter model trained with reinforcement learning outperformed every evaluated 7B open-weights math specialist and remained competitive with six frontier commercial APIs. It also transferred under unchanged prompting to GSM8K, MATH-500, and MMLU-college-math. A human audit of 177 verifier decisions found 97.7% accuracy.

The authors release code, data, verifier modules, and annotations via a public repository, providing a reusable environment for training and evaluating falsification capabilities in mathematical reasoning.