Apple pre-trains first tri-modal masked diffusion model from scratch across text, image-text, and audio-text

Apple’s Machine Learning Research group describes the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data, diverging from prior bi-modal approaches that initialize and fine-tune a base unimodal model.

The team systematically analyzes multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and provides optimized inference sampling defaults. They propose a novel stochastic differential equation (SDE)-based reparameterization that decouples physical batch size—often chosen for compute efficiency—from logical batch size, which is tuned to balance gradient variance during stochastic optimization.

In experiments, the researchers pretrain a preliminary tri-modal model at 3 billion parameters, trained on 6.4 trillion tokens, demonstrating strong performance across text generation, text-to-image (T2I), and text-to-speech (T2S) tasks.

The study characterizes itself as the largest-scale open investigation of multimodal discrete diffusion models to date, aiming to provide actionable insights into scaling behaviors across multiple modalities.