Paper quantifies depth-dependent fragility in MoE layers, finds late layers tolerate aggressive expert masking
A sensitivity analysis of Qwen3.6-35B-A3B shows early and middle MoE layers are highly fragile to expert masking, while late layers tolerate aggressive pruning, enabling targeted compression strategies.
1 source · cross-referenced
- A new arXiv paper analyzes layer-wise sensitivity in a 35B-parameter MoE model (Qwen3.6-35B-A3B) using magnitude-based expert masking on a code translation benchmark.
- Early (0–9) and middle (10–29) MoE layers are highly fragile to masking, while late layers (30–39), especially very-late layers (35–39), tolerate aggressive masking with minimal quality loss.
- Flat all-layer masking at 30% retains only 150 of 300 outputs at 300-prompt scale, whereas late-focused policies retain 249–255 outputs while masking 640–1,145 experts.
- A narrow very-late policy (layers 35–39 @ 50%) achieves the best quality/masking tradeoff on a 500-prompt validation set, retaining 419 of 500 outputs while masking only 640 of 10,240 experts.
- Reducing top-k routing from 8 to 6 experts per token yields wall-clock speedups without quality loss in a small probe, though it does not yet compose cleanly with aggressive masking.
A new arXiv preprint presents a systematic sensitivity analysis of Mixture-of-Experts (MoE) layers in the Qwen3.6-35B-A3B model, a 35 billion parameter architecture with 40 MoE layers, 256 experts per layer, and top-8 routing. The study evaluates magnitude-based expert masking across three evaluation scales—100, 300, and 500 prompts—using the XLCoST cross-lingual code translation benchmark and three H100 GPU servers.
The authors report that layer sensitivity to expert masking is strongly depth-dependent. Early layers (0–9) and middle layers (10–29) are highly fragile, suffering significant quality degradation even under modest masking. In contrast, late layers (30–39), and especially very-late layers (35–39), tolerate aggressive masking of low-magnitude experts with minimal impact on output quality.
At a 300-prompt scale, masking 30% of experts uniformly across all layers retains only 150 of 300 outputs rated Good or Similar. By contrast, late-focused masking policies retain between 249 and 255 outputs while masking between 640 and 1,145 experts. On a held-out 500-prompt validation set, a narrow policy targeting only layers 35–39 with 50% masking achieves the strongest tradeoff, retaining 419 of 500 Good or Similar outputs while masking just 640 of the model’s 10,240 total experts.
The paper also examines reducing the top-k routing width from 8 to 6 active experts per token. In a 100-prompt probe, this change yields a large observed wall-clock reduction without any loss in Good or Similar outputs. However, the authors note that this reduction does not yet compose cleanly with aggressive expert masking strategies, indicating a need for further integration work.
The findings establish an empirical basis for depth-aware MoE expert masking and suggest practical pathways for model compression, including physical weight surgery, activation-based expert scoring, and training-based recovery. These techniques could enable more efficient deployment of large MoE models without substantial quality tradeoffs.
- Aug 17, 2026 · arXiv cs.AI
Paper proposes RubricForge to reduce over-crediting in LLM agent evaluations
Trust79 - Aug 17, 2026 · arXiv cs.AI
Study finds modular cognitive architecture emerging in large language models across 46 tasks
Trust79 - Aug 16, 2026 · Hacker News — AI (100+ points)
Researchers train language models on a K–5 curriculum to study knowledge acquisition boundaries
Trust79