Hugging Face releases NeoMME, a family of multilingual multimodal encoders trained from scratch

Hugging Face introduced NeoMME, a family of multilingual multimodal encoders with 260M and 800M parameters. Unlike many visual language models, NeoMME does not rely on a separate pretrained vision tower or a causal language model. Instead, a single bidirectional Transformer processes both text tokens and raw image patches end-to-end, trained from scratch using a masked discrete-diffusion objective.

The architecture uses native multimodal inputs: text is tokenized with a custom BPE tokenizer (131k-token vocabulary), while images are divided into 32×32 patches and projected via a small MLP. Both modalities share the same computational path within the Transformer encoder. The models support dynamic image resolution, preserving aspect ratio, and a long context length of 16,384 tokens—enough for up to two 4K UHD images. Training leveraged approximately 524 billion packed input tokens, including 290 billion from text-only examples, with a focus on data efficiency using the NorMuon optimizer.

For downstream evaluation, NeoMME was fine-tuned into NeoMME-Retriever using the page-image approach from ColPali, enabling retrieval of document pages as images without OCR preprocessing. NeoMME-Retriever employs a dual-head design: a dense head for mean-pooled embeddings and a late-interaction head that projects each token or patch to a 128-dimensional vector. This design preserves fine-grained local matches while maintaining compactness.

On the ViDoRe v3 benchmark, both NeoMME sizes achieve competitive nDCG@10 scores relative to model size, lying on the Pareto frontier. At a 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second—roughly twice the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from approximately 1.5 MB to 6 kB per page (255× smaller) while retaining over 95% of baseline nDCG@10.

All model checkpoints are released under the Apache 2.0 license and available via Hugging Face Transformers.