Hugging Face Transformers adds native support for llama.cpp GGUF quantized models

Hugging Face’s Transformers library now supports loading GGUF-quantized models directly through a new `gguf_file` argument in `from_pretrained`, enabling users to run quantized checkpoints via the standard Transformers API without additional configuration.

Initial optimizations target Apple Silicon, where Transformers reuses llama.cpp’s ggml kernels to reduce overhead and bring performance closer to native llama.cpp execution while maintaining the Transformers interface.

GGUF models can also be served locally using `transformers serve`, exposing an OpenAI-compatible API endpoint that clients such as Jan or Pi can consume by registering a custom provider with the provided base URL and model identifier.

Benchmarking compares throughput between Transformers and llama.cpp on an M2 Max MacBook Pro (32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0), using llama-bench for llama.cpp and a custom script for Transformers, with results reported as token-generation rate over 128 decoded tokens.

GGUF is a file format developed by the llama.cpp team that packages model weights, tokenizer information, and optional chat templates into a single file, supporting multiple quantization levels such as Q4_K_M, Q5_K_M, and Q6_K, which trade precision for smaller memory footprints.

Publishers including Unsloth, LM Studio Community, and bartowski provide ready-to-use GGUF checkpoints on the Hugging Face Hub, with millions of downloads reported for GGUF models.

To load a GGUF model, users install the latest Transformers main branch and a compatible kernels build, then call `AutoModelForCausalLM.from_pretrained` with the Hub model ID and the specific GGUF filename, after which standard Transformers workflows apply unchanged.