Skip to content
Tools · Nov 9, 2026

Hugging Face introduces HOPE, a method to prune experts in Mixture-of-Experts models while preserving cooperative groups

HOPE, a second-order pruning method, reduces memory usage in MoE models by up to 50% with minimal performance loss, outperforming first-order methods on agentic coding tasks.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • HOPE is a new pruning method for Mixture-of-Experts (MoE) models that accounts for pairwise expert interactions, unlike first-order methods that score experts independently.
  • At 50% pruning, HOPE achieves the best average rank and up to +6.1% gains on agentic coding tasks compared to the next-best pruning method.
  • HOPE avoids latency issues associated with dynamic loading/off-loading of experts and does not require additional training, composing with quantization.
  • The method minimizes pruning error by solving a quadratic program derived from a surrogate objective that captures expert interactions.

Mixture-of-Experts (MoE) models enable state-of-the-art large language models (LLMs) to scale to hundreds of billions of parameters, but they incur a significant memory cost: every expert MLP must occupy memory at inference time, even when most are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, with approximately 95% of its ~250 GB parameter memory footprint consumed by these expert MLPs alone. However, only 8 experts per layer are used for any given token.

Expert pruning is a direct approach to mitigate this memory bottleneck by permanently deleting the least useful experts. Unlike dynamic loading or off-loading, pruning avoids latency issues and does not require retraining, making it a practical solution. The challenge lies in deciding which experts to remove without harming model performance. Existing first-order pruning methods score experts independently, ignoring interactions between them. In reality, experts form cooperative groups that are consistently co-activated, and removing one expert can destabilize a group, especially at high pruning rates.

HOPE (Higher-Order Pruning of Experts) is a second-order method that accounts for pairwise expert interactions. It defines a surrogate objective, Z, which strictly upper-bounds the dominant component of the pruning error. The surrogate captures cross-terms between pairs of experts, representing their interactions, which first-order methods discard. HOPE minimizes the expected value of Z over a calibration dataset, reformulating the problem as a quadratic program (QP) with a continuous relaxation to obtain the optimal prune-set.

At a 50% pruning rate, HOPE achieves the best average rank among pruning methods and delivers up to +6.1% performance gains on agentic coding tasks compared to the next-best method. The method composes with quantization and does not introduce latency overhead, making it suitable for production deployment.

The HOPE objective is derived to minimize pruning error, defined as the change in layer output when experts are removed. The prune-set is selected to reduce this error, ensuring that cooperative groups of experts remain intact. The quadratic program is solved using a calibration dataset to compute the F-matrix, where each entry captures the average gate-weighted output product of a pair of experts across tokens where both are active.

Sources
  1. 01Hugging Face — Your Experts Have Partners. There's HOPE for Keeping Them Together
Also on Tools

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.