Hugging Face releases Beam, a 501B-parameter open Mixture-of-Experts model with 1M-token context

Hugging Face has released Beam, the first model from the US-based lab Reflection, under the Apache 2.0 license. Beam is a sparse Mixture-of-Experts (MoE) model with 501B total parameters, of which 23B are active per token. It supports text-only inputs and outputs and is designed for long-context reasoning and agentic coding workloads.

The model architecture includes 52 decoder layers with a hidden size of 5,120. It employs a hybrid attention mechanism: 43 layers use sliding-window attention over the previous 512 tokens, while 9 layers attend globally to the full context. This design reduces memory growth during inference, with the KV cache growing by approximately 36 KiB per token instead of 208 KiB per token in a fully global setup.

Beam’s MoE block uses 256 experts per layer, each a SwiGLU MLP with an intermediate size of 2,560. The router follows a group-limited scheme, selecting the top 8 experts from the top 4 of 8 groups (32 experts each). Only 8 experts run per token, enabling inference at the speed of a 23B-parameter model while maintaining the capacity of a 501B-parameter model.

The residual stream in Beam uses a depth-dependent scaling scheme: each sublayer output is added to the stream with weight 1/sqrt(k), where k is the layer index, ensuring all sublayer outputs and the input embedding contribute equally to the final hidden state. The model also applies a tanh softcap at ±30 to the final logits.

Beam ships with day-0 support in Transformers, vLLM, and SGLang, and includes Rust-based harness code for performance optimization. The model is available in BF16, FP8, and NVFP4 checkpoints, with the BF16 checkpoint weighing approximately 1 TB in safetensors format.

Benchmark results provided by the Reflection team show Beam’s performance across multiple precisions. On SWEBench Verified, Beam (BF16) achieves 81.6%, while on Terminal Bench v2.1 it reaches 80.1%. Performance is consistent across BF16, FP8, and NVFP4 checkpoints, with minor variations.