Hugging Face’s transformers backend for vLLM now matches native inference speed for supported models

Hugging Face and vLLM jointly announced that the transformers modeling backend for vLLM now achieves throughput comparable to vLLM’s hand-written native implementations for supported model architectures. Model authors can run Hugging Face models in vLLM without porting by adding a single flag, --model-impl transformers, while continuing to use existing parallelism options like tensor, data, and expert parallelism.

The teams evaluated the backend against native vLLM implementations across three Qwen3 variants: a 4B dense model on a single GPU, a 32B dense model using tensor parallelism, and a 235B-parameter FP8 Mixture-of-Experts model using data and expert parallelism on an 8×H100 node. In each case, the transformers backend met or exceeded the throughput of the native implementations.

The performance gains come from runtime graph fusion and static analysis. The backend uses torch.fx to analyze the model’s graph, then applies abstract syntax tree manipulation to rewrite operations in place, fusing them into patterns that align with vLLM’s optimized kernels. Supported fusions include vLLM’s MergedColumnParallelLinear, QKVParallelLinear, and expert parallelization kernels for Mixture-of-Experts models.

The fused models remain fully compatible with torch.compile and CUDA Graphs, and they can be used for training and evaluation in addition to inference. The announcement notes that models using linear attention or custom Hub implementations not written to the transformers compliance standard are not currently supported, but linear attention support is planned.

To enable the backend, users upgrade the vllm package and run vllm serve with the --model-impl transformers flag. The teams provided a reproducible benchmark script to compare configurations, and plan to publish a detailed technical post explaining the fusion methods.