Apple study finds language discrimination improves multilingual speech models

Apple’s Machine Learning Research group describes a method to improve multilingual self-supervised speech models by strengthening the model’s ability to discriminate languages during pretraining. In controlled English/French HuBERT experiments, the team reports that adding language discrimination reduces the multilingual gap on continuous phonetic and higher-level linguistic measures while preserving substantial cross-language sharing.

The researchers tested two interventions: an auxiliary language classifier and per-language k-means targets. With these additions, continuous-feature phone discrimination error (phone-ABX) decreased from 11.6% in the bilingual baseline to 10.4%, approaching the monolingual baseline of 10.8%. Lexical performance (sWUGGY) increased from 52.1% to 56.7%, compared to 58.5% for the monolingual model, and prosodic performance (ProsAudit, lexical subtask) rose from 68.9% to 72.9%, versus 72.6% for monolingual.

The strongest gains on most linguistic measures occurred when language discrimination was introduced in the first training iteration; later or repeated interventions yielded smaller improvements and were accompanied by increased language-wise segregation.

The findings support a causal role for language discrimination in reducing the additional cost of multilingual learning, according to the authors.