LRMoE system reduces MoE training time by up to 44.8% on memory-constrained GPUs
New memory-aware training system recalibrates backward demand and balances workload to cut MoE-layer training time while preserving routed computations.
1 source · cross-referenced
- LRMoE is a memory-aware MoE training system that jointly recalibrates backward demand and balances retained workload during backpropagation.
- Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2–44.8% relative to SYMI.
- The system plans retention quotas and expert placement under resource constraints during forward execution while preserving all routed computations.
- Physical compaction reduces backward traffic, computation, and execution-buffer demand by forming smaller communication payloads and variable-size grouped GEMMs.
Sparse Mixture-of-Experts (MoE) models enable scalable capacity for web applications but suffer from uneven routing that creates computation and communication bottlenecks. Existing workload redistribution methods often exacerbate memory pressure by adding expert copies that compete with token-related buffers for limited GPU memory.
LRMoE, a memory-aware MoE training system, jointly recalibrates backward demand and balances retained workload. During forward execution, it plans retention quotas and expert placement under resource constraints while preserving all routed computations. In backpropagation, a load recalibration metric estimates each token-expert pair’s contribution to the expert update and selects pairs within these quotas.
A capacity-aware routing algorithm then distributes retained pairs among eligible instances of their original experts. Physical compaction forms smaller communication payloads and variable-size grouped GEMMs, reducing backward traffic, computation, and execution-buffer demand.
Evaluations on Qwen3-30B-A3B and GLM4.5-106B-A12B show LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2–44.8% compared to SYMI. The system targets memory-constrained nodes, offering a practical path to more efficient MoE training without architectural changes to the underlying models.
- Nov 11, 2026 · The Verge — AI
Google DeepMind unveils AlphaGenome 2 to predict genetic variant effects on cell types
Trust79 - Nov 11, 2026 · Google DeepMind — Blog
Google DeepMind unveils AlphaGenome 2 to predict DNA variant effects in specific cell types
Trust84 - Nov 11, 2026 · arXiv cs.AI
Researchers demonstrate auditable human-agent teams for autonomous synchrotron X-ray experiments
Trust79