Researchers propose EntropyMoE, a sparse Mixture-of-Experts architecture for tokenizer-free LLMs

Byte-level large language models (LLMs) have recently become more competitive by grouping raw bytes into dynamically sized patches rather than relying on traditional tokenization. However, existing byte-patch architectures apply uniform dense feed-forward computation across all patches, which does not adapt model capacity to variations in patch semantics or granularity.

EntropyMoE, introduced in a new arXiv preprint, replaces these dense feed-forward modules with Top-K expert layers within a global patch Transformer. Each dynamic byte patch serves as the routing unit, and its byte coverage determines its contribution to workload accounting. The router selects experts using patch entropy—the same signal used to construct dynamic patches—as the primary routing coordinate.

The authors report that patch entropy and patch length jointly define the feature space for regulating expert specialization. In experiments, EntropyMoE achieved the lowest held-out bits-per-byte among matched dense and sparse baselines while maintaining comparable downstream accuracy. This establishes patch entropy as an effective signal for sparse conditional computation and extends MoE modeling beyond tokenizer-based representations.

The work is positioned as a step toward more efficient tokenizer-free LLMs, where dynamic patching and entropy-aware routing reduce unnecessary computation on semantically homogeneous or low-information patches.