Apple outlines design axes for stabilizing semi-supervised federated ASR training

Apple’s Machine Learning Research team describes a semi-supervised federated learning (SSFL) approach for automatic speech recognition (ASR) that relies on a small labeled seed dataset on the server and a teacher model to generate pseudo-labels for unlabeled client data.

The authors argue that ASR is especially sensitive to pseudo-label errors, which compound both across output sequences and across training rounds, often leading to divergence and a large performance gap relative to fully-supervised federated learning.

They identify two coupled design axes that determine whether training stabilizes: the teacher axis (which model generates pseudo-labels) and the anchor axis (server-side updates on labeled data that stabilize training).

On the teacher axis, an online per-client teacher—each client’s own evolving model—initially diverges but, once stabilized, matches or outperforms a broadcast global teacher (a fixed server model within a round) in-domain and remains competitive under domain shift. A transitioning teacher strategy, which switches from global to online at a specified round, can match or beat both approaches.

On the anchor axis, the server must continue training on labeled data between rounds; otherwise the online teacher drifts. This interleaved server-side training governs convergence more than the seed model itself.

The two axes are interdependent: aggressive teacher choices only pay off once the anchor stabilizes training, which is highly sensitive to data augmentation and batch size—settings that control the amount of input and gradient noise injected by the server.

The amount of stabilization required depends on the domain, specifically the dispersion of the seed data and its overlap with client data.

The proposed guidelines improve over the strongest prior SSFL method on 9 of 11 evaluation pairs, yielding average gains of 20.8% in-domain and 10.0% cross-domain, and narrowing the gap to fully-supervised federated learning.