Hugging Face introduces HOPE, a method to prune experts in Mixture-of-Experts models while preserving cooperative groups
HOPE, a second-order pruning method, reduces memory usage in MoE models by up to 50% with minimal performance loss, outperforming first-order methods on agentic coding tasks.
1 source · cross-referenced
- HOPE is a new pruning method for Mixture-of-Experts (MoE) models that accounts for pairwise expert interactions, unlike first-order methods that score experts independently.
- At 50% pruning, HOPE achieves the best average rank and up to +6.1% gains on agentic coding tasks compared to the next-best pruning method.
- HOPE avoids latency issues associated with dynamic loading/off-loading of experts and does not require additional training, composing with quantization.
- The method minimizes pruning error by solving a quadratic program derived from a surrogate objective that captures expert interactions.
Mixture-of-Experts (MoE) models enable state-of-the-art large language models (LLMs) to scale to hundreds of billions of parameters, but they incur a significant memory cost: every expert MLP must occupy memory at inference time, even when most are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, with approximately 95% of its ~250 GB parameter memory footprint consumed by these expert MLPs alone. However, only 8 experts per layer are used for any given token.
Expert pruning is a direct approach to mitigate this memory bottleneck by permanently deleting the least useful experts. Unlike dynamic loading or off-loading, pruning avoids latency issues and does not require retraining, making it a practical solution. The challenge lies in deciding which experts to remove without harming model performance. Existing first-order pruning methods score experts independently, ignoring interactions between them. In reality, experts form cooperative groups that are consistently co-activated, and removing one expert can destabilize a group, especially at high pruning rates.
HOPE (Higher-Order Pruning of Experts) is a second-order method that accounts for pairwise expert interactions. It defines a surrogate objective, Z, which strictly upper-bounds the dominant component of the pruning error. The surrogate captures cross-terms between pairs of experts, representing their interactions, which first-order methods discard. HOPE minimizes the expected value of Z over a calibration dataset, reformulating the problem as a quadratic program (QP) with a continuous relaxation to obtain the optimal prune-set.
At a 50% pruning rate, HOPE achieves the best average rank among pruning methods and delivers up to +6.1% performance gains on agentic coding tasks compared to the next-best method. The method composes with quantization and does not introduce latency overhead, making it suitable for production deployment.
The HOPE objective is derived to minimize pruning error, defined as the change in layer output when experts are removed. The prune-set is selected to reduce this error, ensuring that cooperative groups of experts remain intact. The quadratic program is solved using a calibration dataset to compute the F-matrix, where each entry captures the average gate-weighted output product of a pair of experts across tokens where both are active.
- Nov 11, 2026 · Hugging Face
Hugging Face proposes token pooling-aware training to improve compressed multi-vector retrieval
Trust79 - Nov 11, 2026 · TechCrunch — AI
Fractile raises $720M Series C to develop AI chips and compiler stack
Trust79 - Nov 11, 2026 · The Verge — AI
Google’s Tabletop AI experiment lets users build custom collaborative apps via waitlist
Trust79