Apple study compares diffusion and autoregressive language models on performance and scaling trade-offs

Apple’s Machine Learning Research group published a study comparing the performance characteristics of diffusion language models (DLMs) and autoregressive language models (ARMs). The work combines theoretical analysis with empirical profiling to characterize trade-offs between the two approaches.

The researchers note that ARMs, which generate tokens sequentially conditioned on prior tokens, have been the dominant paradigm for LLMs but suffer from low arithmetic intensity due to sequential dependencies in next-token prediction. DLMs, by contrast, generate output tokens in parallel, which can mitigate the limitations of sequential decoding and increase arithmetic intensity.

The study finds that while DLMs can achieve higher arithmetic intensity than ARMs through parallel token generation, they do not scale effectively with longer contexts. To address this, the authors explore block-wise decoding for DLMs, a technique that decouples arithmetic intensity from sequence length and enables better scaling to long contexts, similar to ARMs.

The authors also examine batched inference and report that ARMs exhibit superior throughput because they benefit more from parallelism across sequences in the batch. The study highlights opportunities for accelerating DLM inference, emphasizing that reducing the number of sampling steps is key for open-source DLMs to achieve lower latency relative to ARMs.

The paper includes contributions from researchers affiliated with Seoul National University, UC Berkeley, ICSI, LBNL, and the University of Texas at Austin, and is authored by Minseo Kim, Coleman Hooper, Aditya Tomar, Chenfeng Xu, Mehrdad Farajtabar, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami.