Apple details memory-efficient audio synthesis architecture for on-device Siri Expressive Voices
A decoupled temporal depth diffusion transformer detokenizer converts semantic audio tokens to high-fidelity audio within the tight compute and memory budget of Apple’s AMX, enabling real-time, on-device synthesis with a 16x speedup and 21MB peak runtime memory.
1 source · cross-referenced
- Apple describes a memory-efficient audio synthesis architecture for on-device Siri Expressive Voices powered by AFM 3 Core Advanced.
Apple’s Machine Learning Research team outlines a new architecture for memory-efficient audio synthesis that underpins Siri Expressive Voices. The system converts semantic audio tokens generated by the AFM 3 Core Advanced foundation model into high-fidelity audio entirely on-device, using the Apple Matrix Coprocessor (AMX).
The architecture features a three-component detokenizer design—a streaming encoder, a temporal decoder, and a depth decoder—that decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all residual vector quantization (RVQ) levels autoregressively, replacing the need for dedicated per-level decoders in prior multi-decoder systems.
Deployed on the AMX, the detokenizer achieves roughly 10ms per generation step, approximately 16 times faster than real time, with a peak runtime memory of about 21MB and 329MB of on-device assets. This enables continuous streaming synthesis of audio ranging from 20 to 320 seconds alongside the on-device foundation model.
The design replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches with constant, small-footprint operations. Comprehensive ablation studies validate the effectiveness of key components, including DiT conditioning mechanisms, temporal lookahead processing, and unified depth decoding strategies.
Audio quality assessment through phonetic discriminability analysis, perceptual quality metrics, and neural quality estimation confirms the architecture maintains synthesis fidelity while achieving computational efficiency gains. The system is deployed in production as part of Siri Expressive Voices, supporting features like Pace and Expressivity customization sliders and custom assistant voices.
Operating within a 1-billion-parameter activation size in AFM 3 Core Advanced, the architecture improves Mean Opinion Score (MOS) by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) compared to the prior on-device text-to-speech system.
- Aug 1, 2026 · Apple — Machine Learning Research
Apple researchers propose graph-based sensemaking workflows using UMAP’s internal kNN graph
Trust79 - Aug 1, 2026 · Apple — Machine Learning Research
Apple proposes MoMo, a two-stage imitation-learning framework for robot motion control
Trust84 - Aug 1, 2026 · Microsoft Research
Microsoft Research proposes EvoLib, a framework that lets models learn from experience during inference without fine-tuning
Trust79