Researchers propose TEXAS method to improve Mixture-of-Experts LLM adaptation

Researchers from multiple institutions propose Task-Expert-Aware Supervision (TEXAS), a method for adapting Mixture-of-Experts (MoE) language models to downstream tasks by identifying task-relevant experts from routing patterns. Unlike prior approaches that rely on aggregate usage statistics, TEXAS compares expert activations on instances the base model solves correctly versus those it fails, retaining experts more strongly activated on successful instances.

The method then upweights answer tokens during fine-tuning when they activate these identified experts, enabling the model to specialize without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution. The authors report that across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings and improves over the strongest baseline by 1.3 to 1.5 points on average.

Ablation studies and further analyses are provided to validate both the discovered experts and the supervision strategy, supporting the claim that correctness-conditioned expert identification and token-level supervision allocation are effective for MoE adaptation.