Apple proposes latent-space distillation to compress on-device speech tokenizer by 2.8× with minimal accuracy loss

Apple’s Machine Learning Research team describes a distillation technique that compresses the streaming neural audio encoder used for on-device speech recognition. The encoder acts as a tokenizer, mapping short waveform windows to representations consumed by a downstream language model. Because the foundation model uses Instruction-Following Pruning, only a subset of experts resides in DRAM at any time, leaving the always-on tokenizer to compete for memory bandwidth and capacity.

Rather than distilling to discrete tokens or output distributions, the researchers regress the teacher model’s pre-quantizer latent representations—the shared latent space immediately upstream of both the quantizer and the language-model bridge. A single affine layer absorbs width mismatch between teacher and student, and the same recipe applies to tokenizers pretrained in isolation or jointly trained with a language model.

In experiments across six teacher–student pairs, the distilled student at 2.8× compression stayed within 1.9% relative word error rate (WER) of the teacher without any fine-tuning. Against an independently trained tokenizer of identical capacity, the distilled student improved WER by 3.9% relative.

The authors note the method covers both token interfaces supported by the system-wide dictation pipeline, offering a general compression strategy for on-device speech encoders.