vLLM v0.32.0 adds structured decision endpoint, low-precision KV cache support, and Model Runner V2
The open-source serving framework vLLM reaches version 0.32.0 with 814 commits from 363 contributors, introducing a new system decision endpoint, expanded low-precision KV cache support, and a next-generation model runner.
1 source · cross-referenced
- vLLM v0.32.0 introduces a new POST /v1/systemone endpoint for structured decisions and generic parsers, enabled by default.
- Low-precision KV cache support expands to NVFP4 on SM8x/SM12x/SM90/SM100 and FP8 on SM100, with measured throughput and memory trade-offs.
- Model Runner V2 debuts with GPU n-gram speculator, Elastic EP, pipeline parallelism for MLA models, and becomes default on ROCm.
- Startup and serving optimizations include precompiled FlashInfer kernels, warm autotune tables, shared-memory tensor arena, and load-aware API routing.
- Breaking changes include Python 3.10 deprecation, stricter CORS defaults, and removal of several legacy features.
vLLM v0.32.0 adds a new POST /v1/systemone endpoint that answers structured decision questions (choice, noul, score) over a state, and is enabled by default. The endpoint is also supported on Cloudflare/clef-omni, LiquidAI/d1-3B, and DiffusionGemma.
Low-precision KV cache support is expanded across multiple GPU architectures. NVFP4 KV cache now runs on SM8x and SM12x, SM90, and SM100, and with XQA decode on SM12x. GLM-5.x defaults to an FP8 KV cache on SM100, with benchmarked trade-offs of 2.3–5.5% more throughput at a 7–11% time-to-first-token cost. On SM90, fp8_ds_mla delivers up to 31% more output throughput for GLM-5.3-Flash. ROCm gains a 4-bit ultraquant_4bit KV cache with roughly twice the density of FP8.
Model Runner V2 introduces a GPU n-gram speculator with variable-length drafts, Elastic EP, pipeline parallelism with prefill context parallelism for MLA models, a stock torch.compile mode, and becomes the default on ROCm with MRV1 exceptions removed.
Startup and serving optimizations include vllm download-kernels installing precompiled FlashInfer kernels, vllm preload --preload-autotune keeping the FlashInfer autotune table warm across restarts, a shared-memory tensor arena broadcasting large multimodal tensors to workers with one copy, and tokenizers and processors loading from local snapshots. With --api-server-count above 1, each connection is routed to the least-loaded API worker, delivering 43–85% more requests per second on short requests in internal benchmarks.
New and updated model support includes EmbeddingGemma 2 omni-modal embeddings, MiniCPM-V 4.7, Nemotron 3.5 ASR, and video input in the Transformers modeling backend. GLM-5.3 benefits from sparse MLA index conversions and fused multi-step draft decode. HiSparse per-request residency caching improves throughput by up to 58% on GLM-5.3-NVFP4 on B200 with long, fully GPU-resident contexts at high concurrency.
Security fixes address resource exhaustion in chat templates, tokenizer max_token_id issues, and P2P ZMQ socket exhaustion. Default CORS origins are restricted, untrusted images are limited to raster formats, shared-memory multimodal cache handles are signed, and JSON schemas nested deeper than 64 levels are rejected.
Breaking changes include dropping Python 3.10 support, restricting default CORS origins, requiring Transformers >=5.16.1 and <5.20.0 and tokenizers >=0.23.2, removing torch.compile sequence parallelism and AsyncTP passes, removing cascade attention and mamba_cache_mode="all", no longer reserving max_tokens from the context window, and renaming Responses API reasoning stream events.
Release artifacts include Python wheels for CUDA 12.9 and 13.0, ROCm, CPU, and XPU, as well as Docker images for CUDA 12.9/13.0 (including Ubuntu 24.04 variants), ROCm, CPU, and XPU.
- Nov 11, 2026 · Hugging Face
Hugging Face proposes token pooling-aware training to improve compressed multi-vector retrieval
Trust79 - Nov 11, 2026 · TechCrunch — AI
Fractile raises $720M Series C to develop AI chips and compiler stack
Trust79 - Nov 11, 2026 · The Verge — AI
Google’s Tabletop AI experiment lets users build custom collaborative apps via waitlist
Trust79