Apple proposes method to detect code-switching in speech without transcription

Apple’s Machine Learning Research team describes a method to detect code-switching—alternating languages within a single utterance—directly from speech audio, bypassing the need for transcription. The technique leverages language-selective hidden-state dimensions in pretrained speech encoders, aggregating frame-level activations to predict whether both languages are present in an utterance.

To operationalize the approach, the team first identifies language-selective dimensions using monolingual recordings, then applies a small code-switching set to tune a decision rule. Evaluated on CS-FLEURS across 14 language pairs, the Whisper-small encoder achieves a mean F1 of 0.798, compared with 0.776 for a CNN baseline and 0.751 for a HuBERT linear baseline trained for monolingual language identification.

The authors argue that pretrained speech encoders already encode sufficient information to identify code-switching utterances, enabling a lightweight tool for building more representative multilingual corpora without the overhead of accurate multilingual transcription.

The work is situated within broader efforts to improve multilingual speech processing, where code-switching data scarcity limits model performance. Related Apple research includes an iterative pseudo-labeling approach for Mandarin-English code-switching ASR and prior work on end-to-end speech translation for code-switched speech.