Hugging Face Transformers v5.20.0 adds native support for six new models and introduces breaking changes

Hugging Face Transformers v5.20.0 adds native support for six new models, each targeting distinct use cases. Muse Spark Open, a multimodal mixture-of-experts model from Meta, supports text, images, audio, and video inputs with 816B total parameters (42B active per token) and a shared context window of up to 1,048,576 tokens. It is optimized for agentic coding tasks such as multi-file editing and debugging, with checkpoints available in BF16 and NVFP4 formats.

Ling 3.0 Flash, a hybrid mixture-of-experts language model from InclusionAI, integrates natively via the BailingHybridForCausalLM architecture. Its decoder alternates between Kimi Delta Attention and Multi-head Latent Attention layers, using interleaved rotary embeddings and a head-wise output gate. The model does not require trust_remote_code=True for loading.

Kolibri 1, developed by Aleph Alpha, is a mixture-of-experts reasoning model focused on German and English. It routes every token to six of 384 experts per layer using sigmoid routing, with four sliding-window attention layers alternating full-attention layers and RoPE applied only in sliding-window layers.

MiniCPM-V 4.7, an update to the MiniCPM-V series from OpenBMB, introduces canvas M-RoPE to preserve spatial relationships in images and temporal order in videos. It pairs a SigLIP vision encoder with a Qwen3.5 language model backbone and supports 4x and 16x visual downsampling modes.

UnlimitedOcr, a 3B-parameter OCR model from Baidu, features Reference Sliding Window Attention to maintain a constant KV cache during decoding. This enables transcription of dozens of pages in a single forward pass, with support for 'gundam' and 'base' modes and region detections with bounding boxes.

UltraBERT, a family of bidirectional text encoders from an unnamed team, supports sequences up to 32,768 tokens. It uses rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention, with released checkpoints including UltraBERT-150M, UltraBERT-380M, and UltraBERT-Embed-150M.

MOSS-Transcribe-Diarize 0.9B, an end-to-end audio model, combines a Whisper-style encoder with a Qwen3 language model to perform multi-speaker transcription, diarization, and timestamping in a single pass. It supports long-form audio via Whisper-window chunking, promptable transcription with hotword biasing, and over 50 languages.

The release also introduces breaking changes, including inference speed regressions with EP, removal of deprecated cache utilities affecting models like minimax_m3_vl and pi0, and stricter sub-config initialization via SubConfigSpec. The minimum kernels version is now 0.17.2, enabling fallback to original implementations when kernel loading fails.