Hugging Face adds private datasets to Open ASR Leaderboard to combat benchmarking gaming

The Open ASR Leaderboard, which has received over 710,000 visits since September 2023, now incorporates private evaluation datasets to reduce model overfitting to public benchmarks. Hugging Face collaborated with data providers Appen Inc. and DataoceanAI to curate high-quality speech datasets spanning multiple English accents and speaking styles.

The new private dataset includes approximately 29 hours of audio across Australian, Canadian, Indian, US, and British English accents. Each accent variant exists in both scripted and conversational forms, with datasets ranging from 1.02 to 8.82 hours. The datasets are intentionally withheld from model developers to prevent explicit optimization against the test sets.

The leaderboard implements a structural safeguard: the default Average WER metric excludes private dataset results, preventing private data from immediately affecting model rankings. Developers can optionally toggle private data visibility to observe downstream effects, but the primary leaderboard ranking remains based on public datasets. This design choice prioritizes transparency while protecting evaluation integrity.

The update introduces fine-grained performance metrics that break down results by speech style (scripted versus conversational) and accent region (US versus non-US). These targeted metrics are computed as macroaverages across data providers to prevent developers from gaming specific subsets. Hugging Face acknowledges that models trained on data from the private providers may retain some advantage, mitigated by the presence of multiple independent data sources and openness to adding new providers.