Apple paper argues discrete diffusion methods fail to model token dependencies

Apple Machine Learning Research published a paper titled “Limits of Confidence in Diffusion” that analyzes the theoretical and empirical limitations of discrete diffusion methods, including remasking and uniform-state samplers. These methods generate sequences by writing multiple token positions per step, sampling each from a per-position distribution and selecting which positions to update from those same distributions.

The authors argue that a sampling step matches the training distribution only when the positions written are conditionally independent given the tokens already fixed. They prove that no product of per-position distributions can match a dependent group of tokens, even when the marginal distributions are identical. This means two joint distributions can share the same per-position marginals while differing in which value combinations occur, making per-position distributions insufficient to capture dependencies.

To validate their theoretical findings, the researchers tested the claims on ScanAndAdd, a synthetic task with a known closed-form joint distribution. They report that every group of two or more undetermined positions written by a confidence ranking is dependent, and that the generated distribution is 29 times the sampling-noise floor total variation. Per-sample metrics are reported as 1.0.

The paper concludes by proposing that valid sampling steps must satisfy a conditional independence requirement, implying that current discrete diffusion approaches may need architectural or procedural changes to properly model token dependencies in domains such as pixels, phonemes, or words.