Meta releases Muse Spark Open, an 816B-parameter open-weight multimodal MoE model for coding agents

Meta released Muse Spark Open, an open-weight frontier multimodal mixture-of-experts model. The model has approximately 816 billion total parameters, with around 42 billion active per token, and supports a context window of up to one million tokens.

Muse Spark Open accepts text, images, audio, and video as inputs and generates text, providing a common interface for developers to query across different input types. Meta trained the model for agentic coding workflows, including reading codebases, writing and debugging code, and calling tools. It can also process screenshots, voice notes, or screen recordings while working on code.

The model’s architecture includes 62 layers with a hidden dimension of 8192, 128 query heads, and 16 key-value heads using grouped-query attention. Each layer applies QK-norm and a learned residual gate. The mixture-of-experts block contains 256 experts, routing 8 experts per token; experts hold approximately 97% of total parameters, enabling faster inference at the ~42B-parameter compute level while requiring the memory footprint of an 816B-parameter model.

Muse Spark Open includes a 1.8B-parameter perception encoder for images and video, splitting each image into 14×14 patches and producing 4096-dimensional features. The encoder uses learned absolute embeddings, 2D RoPE, LayerNorm, GELU, and a mix of full and sparse attention layers. Video input is processed through the same encoder as images.

The audio encoder converts audio into tokens at 12.5 tokens per second, marking audio positions with special tokens. The encoder uses a convolutional pyramid with patch embedders and 1D convolutional blocks across multiple patch sizes, ending with RMSNorm, linear projection, and RMSNorm.

The model supports a 1M-token context window at inference, with max_position_embeddings set to 1,048,576. Grouped-query attention reduces KV cache size; each token adds about 250 KB in BF16, totaling roughly 266 GB for a full 1M-token context.

Meta provides BF16 and NVFP4 checkpoint formats. The BF16 checkpoint is about 1.5 TB in safetensors format and runs across 4 nodes of 8 H200 GPUs with tensor parallelism (TP=32). The NVFP4 checkpoint targets NVIDIA Blackwell GPUs and runs on a single B200 node.

For deployment, Meta offers guidance for llama.cpp, vLLM, and SGLang, and fine-tuning support via TRL. The implementation includes MuseSparkForConditionalGeneration for text generation from multimodal inputs and MuseSparkProcessor for preparing conversations, with distributed configuration for tensor and expert parallelism.