Hugging Face releases Olmo-core 3, an open training framework for trillion-parameter MoE models

Hugging Face published Olmo-core 3, an open training framework designed to scale mixture-of-experts (MoE) models into the trillion-parameter range while maintaining computational efficiency. The release is positioned as part of the stack behind the next generation of Olmo models and continues the lab’s commitment to open training infrastructure.

The upgrade replaces a fully sharded data-parallel system with a distributed data-parallel design that keeps experts resident on GPUs and routes relevant data to them, avoiding repeated weight gathering. In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 tokens/sec/GPU using the earlier implementation—about 2.7× faster.

Olmo-core 3 combines expert parallelism, pipeline parallelism, and a distributed optimizer to split the model and its training state across GPUs. Additional optimizations include rowwise expert parallelism for direct placement of routed data, GPU-resident routing to avoid copying metadata to CPUs, and grouped GEMM to execute many small expert computations efficiently.

The framework also supports MXFP8, a lower-precision format, which in a controlled benchmark on four NVIDIA B300 GPUs increased end-to-end training throughput by about 21% and reduced peak active memory from 103 GiB to 95 GiB relative to BF16. Most gains came from feed-forward computation and inter-expert data movement rather than attention.

Benchmarks reported include a 1.2-trillion-parameter configuration with 58.36 billion active parameters per token across 512 GPUs, achieving a highest observed throughput of 858 TFLOP/s/GPU on NVIDIA B300 GPUs. A separate short-capacity test reached a 2.38-trillion-parameter configuration using DeepEP v2 for expert communication.

The release includes a technical report and interactive demo, and cites experiments that surfaced pitfalls such as "token gerrymandering"—where a routing score intended to encourage balance can mask actual imbalance—and evidence that lowering experts’ learning rates did not improve results in the tested model family.