Researchers propose Attention-Aware Routing to improve Mixture-of-Experts models by coupling routing with attention mechanisms

Researchers from an unspecified set of institutions propose Attention-Aware Routing (AAR), a method that augments the standard router in Mixture-of-Experts (MoE) language models with temporal and spectral features extracted from a sliding window of attention weights. Unlike conventional routing, which relies solely on a token's hidden state, AAR leverages a summary of the model's contextual state that is disentangled from the hidden state.

The authors keep the base transformer entirely frozen and train only the routing parameters, isolating routing as the sole variable under study. On the GSM8K benchmark, AAR achieves a +3.37 percentage point improvement over a routing-only supervised fine-tuning (SFT) baseline on the OLMoE model.

The study further demonstrates that routing and attention form a coupled circuit: changes in routing at layer l propagate through the residual stream to amplify attention sinks at layer l+1, thereby reshaping attention patterns without any direct modification to the attention mechanism itself.

AAR also reduces the tendency for incorrect answers to diverge in length, with incorrect generations becoming shorter, while correct answers maintain their original length. The method exhibits strong depth sensitivity: applying AAR indiscriminately across all layers can degrade factual retrieval performance, whereas mathematical reasoning gains persist when AAR is introduced at deeper layers. This reveals a retrieval–reasoning tension across network depth and suggests that layer-selective AAR can serve as a controlled probe of routing-relevant information carried by attention at different depths.