Apple study quantifies repetition limits in mixture pretraining for language models under data constraints

Apple’s Machine Learning Research team published a study titled 'Scaling Laws for Mixture Pretraining Under Data Constraints,' which examines how to balance scarce target data with abundant generic data during language model pretraining. The research analyzes more than 2,000 language-model training runs spanning multiple model sizes, target dataset sizes, and data types, including multilingual, domain-specific, and quality-filtered mixtures.

The study identifies a fundamental trade-off in mixture pretraining: too little target data in the mixture underexposes the model to the target domain, while too much target data leads to excessive repetition, diminishing returns, and overfitting. Across all settings, the researchers find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training.

According to the paper, scarce target corpora can be reused 15–20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. The authors introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing this scaling law provides a principled way to compute effective mixture configurations, yielding practical recommendations for pretraining under data constraints.

The research builds on prior work from Apple’s ML Research, including a 2025 NeurIPS paper on scaling laws for optimal data mixtures and an ICML paper on scaling laws for forgetting during fine-tuning with pretraining data injection.