Systematic study compares diffusion language models to next-token LLMs across eight benchmarks

Researchers from the University of Modena and Reggio Emilia present the first systematic experimental analysis of modern diffusion language models (DLMs), evaluating eight state-of-the-art DLMs across eight benchmarks that span reasoning, coding, translation, knowledge, and structured problem solving. The study explicitly considers both generation quality and computational efficiency, addressing a gap in prior work where inconsistent evaluation protocols and hyperparameters made cross-model comparisons difficult.

The authors analyze the impact of key inference-time factors—including denoising steps, context length, block size, and parallel unmasking strategies—on model performance and efficiency. They complement large-scale experiments with controlled comparisons of smaller models trained under identical conditions, enabling more granular insights into architectural and scaling choices.

The study finds that DLMs’ behavior is strongly influenced by generation-time design choices, leading to distinct trade-offs between performance and computational cost. These findings provide practical guidance for researchers and practitioners considering DLMs for deployment, highlighting where they may offer advantages over next-token autoregressive models and where they currently fall short.

The paper positions DLMs as an emerging alternative to autoregressive LLMs, noting that DLMs generate text via iterative denoising and allow parallel refinement of entire sequences, unlike next-token prediction. By systematically evaluating modern DLMs across standardized benchmarks, the work contributes empirical evidence on their capabilities and deployment characteristics.