Apple proposes DACA-GRPO to improve reinforcement learning for diffusion language models

Apple’s Machine Learning Research team introduced DACA-GRPO, a method designed to improve reinforcement learning for diffusion language models by addressing two identified weaknesses in current RL approaches. The first weakness is the lack of temporal credit assignment across the denoising trajectory, where existing methods treat all denoising steps as equally important. The second is the systematic bias introduced by mean-field likelihood estimates used for policy optimization, which can lead to suboptimal training dynamics.

DACA-GRPO proposes two complementary mechanisms to mitigate these issues. Denoising Progress Scores extract per-token importance weights from intermediate predictions without requiring additional forward passes. Stratified Masking Likelihood partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias that arises when tokens are treated independently.

The authors evaluate DACA-GRPO on top of three GRPO base methods across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation. The method reports consistent improvements, including gains of up to 5.6 percentage points on math reasoning, 7.4 percentage points on code generation, 36.3 percentage points on constraint satisfaction, and 5.9 percentage points on JSON schema adherence.

The paper frames DACA-GRPO as a lightweight, plug-and-play enhancement that can be integrated into any GRPO-style trainer without requiring changes to the underlying model architecture. The authors note equal contribution from multiple researchers and include institutional affiliations, including a co-author affiliated with The Ohio State University, indicating collaborative academic and industry research.

The work is presented alongside related Apple ML Research publications on GRPO variants, suggesting a broader research program focused on reinforcement learning methods for language models. The publication date listed is September 2026, indicating a forward-looking research disclosure.