Hugging Face and VoiceArena launch open ASR evaluation sets for Hindi and Indian English

Hugging Face and VoiceArena launched Hindi and Indian English evaluation sets on the Open ASR Leaderboard, marking the first Global South languages on the multilingual leaderboard. The new splits include public and private test sets for both languages, designed to expose failure modes that aggregate WER can obscure.

The Hindi and Indian English sets were built to vary along nine axes—geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts—so that models cannot hide disparities behind a single average score. Each split is speaker-disjoint and includes 12 speaker attributes per clip, with 4,888 total speakers across the four splits.

The public and private splits for Indian English total 5.62 hours and 5.58 hours, respectively, while the Hindi splits total 1.33 hours (public) and 4.47 hours (private). Clips average roughly 6–10 seconds, and the datasets span hundreds of districts and dozens of device models to avoid overfitting to specific hardware or locales.

The Hindi sets use a lattice of accepted orthographic variants instead of a normalizer, since Hindi spelling variation cannot be resolved by a fixed mapping. The Indian English sets rely on a normalizer to collapse spelling variants. Both languages emphasize conversational, spontaneous speech with disfluencies included.

Speaker diversity is a core design goal: more than half of all speakers appear only once, and the top ten contributors account for no more than 6.8% of total duration in any split. Recordings span 289 to 814 distinct cities and 18 to 25 device manufacturers, with no single device model exceeding 2.1% of segments in any subset.