Allen AI releases EMO, a modular mixture-of-experts model that enables selective expert use with minimal performance loss

Allen AI has released EMO, a 128-expert mixture-of-experts model designed to achieve modular specialization without relying on predefined semantic categories. The model trains 14 billion total parameters with 1 billion active per forward pass on 1 trillion tokens. Unlike standard MoE architectures where each token independently selects experts, EMO constrains all tokens within a document to activate experts from a shared pool—using document boundaries as a weak supervisory signal that encourages domain expertise to emerge organically from training data.

The core innovation addresses a fundamental limitation in existing MoE systems: standard expert selection produces specialists in low-level lexical patterns (prepositions, punctuation) rather than coherent domains, making small expert subsets unreliable. EMO's document-level routing enforces consistent expert usage across documents, allowing recurring expert groups to form directly from the training signal. The router computes average expert preferences across a document, selects the most-used experts as the document's shared pool, and allows different documents to use different pools.

Implementing the approach required solving two technical constraints. First, load balancing: standard local load-balancing within micro-batches conflicts with EMO's objective of keeping expert usage coherent within documents. EMO applies load balancing globally across many documents, making the two objectives complementary—tokens within documents concentrate expertise while different documents collectively activate all experts. Second, document pool size is randomly sampled during training rather than fixed, preventing overfitting to a single subset size and enabling flexible inference-time expert selection.

Evaluation shows EMO matches standard MoE performance on general benchmarks when all experts are active. More significantly, task-specific expert subsets—constructed by ranking experts by their routing usage on validation data—retain performance with minimal degradation. EMO loses approximately 1% performance when using only 25% of experts (32 of 128 total), and maintains near full-model performance at 12.5% activation (16 experts). This contrasts sharply with standard MoE models, which show severe degradation under selective expert use.

Allen AI is releasing pretrained model weights, a technical report, code, and an interactive visualization of expert specialization through standard channels: Hugging Face model collection, the project's GitHub repository, a dedicated paper, and a web-based visualization tool.