Hugging Face study finds benchmark optimization inflates ASR model scores

Hugging Face introduced a methodology to quantify benchmark optimization in automatic speech recognition (ASR), a phenomenon where models learn benchmark-specific patterns rather than improving at the underlying task. The research team evaluated 11 widely used open-source ASR models and found that several of the highest-scoring systems reproduced benchmark transcripts even when the audio contradicted them or relevant words had been silenced.

In a case study using the VoxPopuli English dataset, the team used a consensus disagreement probe to test whether models transcribed what the audio said or reproduced the benchmark's incorrect reference transcript. An ensemble of independent models with low phoneme error rates was used to flag cases where models unanimously disagreed with the benchmark's reference. For example, six of the 11 models omitted the audible phrase "Thank you, Mr. President" to match the benchmark's erroneous transcript. The same behavior often weakened or disappeared when the content was presented in newly collected voices from EU parliamentary recordings or generic voices.

The study found that models exhibiting benchmark-optimized behavior reproduced erroneous reference transcripts 18–30% of the time. Models with the lowest reported VoxPopuli word error rates were also the most likely to reproduce these errors, as shown in a scatterplot comparing WER with the rate of reproducing incorrect references. When audio clips were resynthesized in a generic text-to-speech voice unconnected to any parliamentary recording, all eleven models restored the omitted courtesy phrase, further indicating that the issue stems from benchmark-specific cues rather than general transcription ability.

The research also introduced a masked entity retrieval test, where numbers were deliberately silenced in audio samples. Since the numbers were absent from the audio, models should not output any number, yet some models produced exact numbers anyway, suggesting reliance on benchmark patterns rather than audio content.