Apple demonstrates few-step text generation with a 1.7B-parameter flow model trained on 2.1 trillion tokens

Apple’s Machine Learning Research group describes Categorical Flow Maps (CFMs) as a non-autoregressive alternative to traditional language modeling, arguing that continuous diffusion and flow matching methods can offer accelerated sampling and other benefits previously limited to continuous modalities like images.

The team trained a 1.7-billion-parameter base flow model on 2.1 trillion tokens and then self-distilled it into a CFM capable of generating diverse, high-quality text in as few as four inference steps while maintaining near-data-level token entropy.

To evaluate CFMs rigorously, the researchers introduce a likelihood bound for CFMs in the semi-discrete setting and demonstrate that CFMs can be used to score models on standard language modeling benchmarks, achieving results comparable to discrete diffusion baselines.

The paper also details practical challenges encountered during large-scale training—such as loss weighting and time scheduling—and provides prescriptive guidance for practitioners aiming to replicate or extend these results.

The authors note that prior work had only evaluated CFMs at modest scales (less than 1 billion parameters), leaving scalability an open question. This study addresses that gap by pushing into the multi-billion-parameter and multi-trillion-token regime.