Hugging Face Transformers v5.22.0 adds support for Mistral Large 4 and FP8 adapter training
The update introduces native integration for Mistral’s new Mixture-of-Experts model and enables trainable FP8 quantization for adapters, alongside bug fixes and performance improvements.
1 source · cross-referenced
- Hugging Face Transformers v5.22.0 adds native support for Mistral Large 4, a 1T-parameter multimodal MoE model with 52B active parameters.
- The release enables FP8 quantized adapter training with backward passes, reducing peak memory in MoE layers from ~28.9 GiB to ~2.8 GiB at 8k tokens.
- Bug fixes include processor warnings, boolean padding in feature extractors, and improved context handling for RoPE.
Hugging Face Transformers v5.22.0 integrates Mistral Large 4, Mistral’s largest and most capable model to date, as a natively supported model in the library. The model is a 1 trillion-parameter Mixture-of-Experts architecture with 49 billion active parameters per token (52 billion including embeddings and output layers), trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s European datacenters.
The update also introduces support for FP8 quantized adapter training, enabling backward passes on dense and grouped FP8 matrix multiplications. This reduces peak memory usage in MoE layers during backward passes from approximately 28.9 GiB to 2.8 GiB at 8k tokens, improving efficiency for fine-tuning workflows.
Additional improvements include fixes for processor-related warnings, boolean padding in the ClapFeatureExtractor, and enhanced handling of context for RoPE (Rotary Position Embedding) via the PermuteForRope utility.
The release resolves several bugs, such as unnecessary processor warnings, incorrect boolean padding in feature extraction, and issues with uninitialized layer cropping. It also updates documentation and removes deprecated benchmarking infrastructure to streamline future development.
Mistral Large 4 unifies instruction following, reasoning, coding, and agentic workflows in a single set of weights and supports a 1M context window. It accepts both text and image inputs with text output, and is natively fluent in over 160 languages, including all official EU languages.
- Nov 11, 2026 · Hugging Face
Hugging Face proposes token pooling-aware training to improve compressed multi-vector retrieval
Trust79 - Nov 11, 2026 · TechCrunch — AI
Fractile raises $720M Series C to develop AI chips and compiler stack
Trust79 - Nov 11, 2026 · The Verge — AI
Google’s Tabletop AI experiment lets users build custom collaborative apps via waitlist
Trust79