Hugging Face Sentence Transformers v6.0 adds MultiVectorEncoder for ColBERT-style late interaction retrieval

Hugging Face’s Sentence Transformers library introduced a new model type in v6.0: MultiVectorEncoder, designed for ColBERT-style late interaction retrieval. This update includes a complete training approach for multi-vector models, enabling users to fine-tune existing models or train new ones from scratch.

The training workflow covers core components: model selection, dataset preparation, loss functions, training arguments, evaluators, and the Trainer class. The library supports fine-tuning an existing multi-vector model such as LightOn’s mLateOn-unsupervised checkpoint, while allowing users to adjust document length limits that are often capped in released checkpoints.

The post highlights the importance of domain-specific fine-tuning for multi-vector models, noting that general-purpose retrievers often truncate long documents and average away fine-grained signals. For example, the author reports that truncation can reduce NDCG@10 by up to 0.24 on a medical retrieval evaluation with passages averaging 941 tokens.

The author demonstrates training a multi-vector model, multi-vector-encoder/mLateOn-medical, in 14.5 hours on a single RTX 3090 GPU. The fine-tuned model outperformed general-purpose retrieval models—dense, sparse, lexical, and multi-vector—on the author’s medical retrieval evaluation.

The training approach allows users to configure document length according to their data needs, addressing a limitation of many released retrieval models that truncate at 180–512 tokens. The post also notes that similar domain-specific fine-tuning has been applied in code retrieval, where LightOn trained LateOn-Code due to limitations of general-purpose late-interaction models.