Apple proposes Path-Constrained Mixture-of-Experts to reduce statistical inefficiency in sparse MoE models

Apple’s Machine Learning Research team proposes viewing sparse Mixture-of-Experts (MoE) computation through the lens of expert paths—the sequence of expert selections a token makes across all layers. The authors observe that, despite N^L possible paths for N experts across L layers, tokens in practice cluster into a small fraction of paths aligned with linguistic function, leaving the vast majority unexplored and representing a statistical inefficiency.

To address this, the team introduces Path-Constrained Mixture-of-Experts (PathMoE), which shares router parameters across blocks of consecutive layers. This constraint amplifies emergent path structure by producing more concentrated path clusters, improving cross-layer consistency, and increasing robustness to routing perturbations.

The researchers evaluate PathMoE on 0.9B and 16B parameter models, reporting consistent improvements in perplexity and downstream tasks compared to independent routing baselines. Notably, PathMoE eliminates the need for auxiliary losses typically used to encourage desired routing behavior.

The work situates PathMoE as a complementary design axis to existing independent routing mechanisms, arguing that expert paths constitute a useful design axis for MoE architectures. The authors include related work on Omni-Router for speech recognition, highlighting broader applications of shared routing decisions in MoE systems.