Hugging Face adds multi-vector embedding models to sentence-transformers library

Hugging Face’s sentence-transformers library, a Python framework for embedding and reranker models, added support for multi-vector (late interaction) embedding models in its v6.0 update. The new MultiVectorEncoder model type enables ColBERT-style retrieval, where documents are encoded as matrices of token-level embeddings rather than compressed into single vectors.

Unlike dense embedding models that compress entire texts into fixed-size vectors, multi-vector models retain one vector per token, typically projected to 128 dimensions. This preserves fine-grained matching information, such as rare entities or exact identifiers, which can be averaged away in single-vector approaches. For example, a query like 'green sofa with wooden legs and rounded cushions' can match each requirement to distinct tokens in a document, rather than blending them into a single vector.

Scoring uses the MaxSim operator, which computes the sum of the maximum cosine similarities between each query token and any document token. This allows for soft alignment between query and document tokens, including synonyms or paraphrases that lexical methods like BM25 would miss. The operator’s output range is bounded by the number of query tokens, reflecting the cumulative similarity scores.

The library supports loading checkpoints from PyLate, Stanford-NLP ColBERT, and colpali-engine, integrating them into the same API used for dense, sparse, and reranker models. This lowers the barrier to adoption for developers already familiar with the library’s workflows.

The primary trade-off is index size. Storing one vector per token increases storage requirements substantially. For instance, encoding 4,874 passages from the Natural Questions dataset with the lightonai/LateOn model produced 608,414 token vectors, averaging 124.8 vectors per passage. The raw storage for these vectors was 311.5 MB in float32 format, compared to 7.5 MB for a dense MiniLM model’s index. Compression methods like PLAID can reduce this overhead by storing centroid IDs and quantized residuals, bringing the index size down to 92 MB for the same data.

The update also includes guidance on installation, model inspection, encoding queries and documents, and integrating these models into semantic search, retrieval-augmented generation, and visual document retrieval pipelines. The blog post provides example scripts and links to documentation for further implementation details.