Few prompt tokens drive most F1 loss when reducing experts in MoE prefill

A new arXiv paper identifies that reducing routed experts during the prefill phase of mixture-of-experts (MoE) models disproportionately harms short-answer question answering performance at specific prompt positions. The study evaluates four MoE families and finds that, at half the experts, three of the families suffer most of their F1 loss due to changes in answer formatting rather than answer correctness. In Qwen3-30B-A3B, the two chat-template tokens immediately preceding the answer account for most of the loss, while in DeepSeek-V2-Lite, the first 8–32 prompt tokens—where the instruction resides—contribute about a fifth of the damage.

The authors use token-, layer-, and attention-path interventions to isolate the effect, showing that reducing experts changes the first token’s attention mass, norm, and largest activation by less than 4%. Attempts to recover performance by restoring super experts or massive activations recover at most about a quarter of the loss, which is substantially less than keeping the sensitive positions at full expert routing. The paper also reports that placing just 4% of tokens at full experts on the task interface keeps F1 within one point of the unmodified model, whereas placing them randomly in the documents results in about 3 points worse performance.

The study further notes that the sensitive tokens consistently appear at prompt ends in every sensitive layout observed, and that DeepSeek’s instruction-related loss disappears when the instruction moves away from the start of the prompt. Time to first token improves by about 14% on 11K-token prompts but shows no significant change on 2K-token prompts.