New training objective improves LLM consistency and performance by aligning validators with frequency-corrected generators

A new preprint introduces FCPA, a training objective designed to align a model’s validator head with a frequency-corrected view of its own generator outputs. The work targets the generator-validator (G-V) gap, a phenomenon where LLMs generate responses that they later deem invalid when asked to validate them.

The authors argue that naive G-V consistency is unreliable because generators often assign low likelihood to valid strings simply due to their a priori unlikelihood. They formalize a correction under a model of rational agents answering questions with multiple valid responses, showing that frequency-corrected G-V consistency emerges as a natural objective.

The proposed method, FCPA, implements this correction during training. In experiments, training with FCPA substantially improves both G-V consistency and generator performance relative to prior approaches. Reported gains include up to a +27 percentage-point increase in Pearson correlation on IFEval and HumanEval benchmarks.

The paper also reports that validator quality is preserved across all evaluated tasks, indicating that the alignment objective does not degrade the validator’s discriminative capabilities.