Apple proposes IDEA Prune, an integrated enlarge-and-prune pipeline for pretraining generative language models

Apple’s Machine Learning Research team describes IDEA Prune, an integrated enlarge-and-prune pipeline for generative language model pretraining. The approach explicitly incorporates enlarged model pretraining—an often-overlooked stage in prior pruning pipelines—into a unified system that also includes pruning and recovery.

The proposed pipeline uses a single cosine annealing learning rate schedule to jointly govern enlarged model training, pruning, and recovery. This design aims to mitigate knowledge loss that can occur when learning rates rise in naive enlarge-and-prune pipelines, while enabling capacity redistribution among surviving neurons.

The authors introduce an iterative structured pruning method for gradual parameter removal, which they argue facilitates smoother compression and improved performance of the final pruned model.

In experiments, the team compresses a 2.8-billion-parameter model to 1.3 billion parameters using up to 2 trillion tokens during pretraining. They report that the integrated approach not only improves token efficiency relative to training target-size models from scratch, but also achieves superior performance in the resulting pruned models.

The paper includes authors affiliated with Apple and academic institutions, including Georgia Institute of Technology and University of Texas at Austin. It is presented as part of Apple’s ongoing research in machine learning, with related prior work such as UPSCALE and PDP also cited.