Transformers adds ggml kernels to run quantized GGUF models
Quantized checkpoints load into PyTorch workflows, initially targeting Apple Silicon and Qwen3.5; missing quantization kernels can expand weights in memory.
from_pretrained selects a quantized checkpoint through gguf_file. Hugging Face's September 22, 2026 announcement adds ggml operations through the kernels library. Its loading example is:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(
model_id,
gguf_file=filename
)
The initial target is Apple Silicon with the Qwen3.5 architecture. At publication, installation requires the development version of Transformers and a PyTorch version compatible with the available kernel builds. Reading the format and accelerating a particular model are separate compatibility questions.
The fallback behavior deserves attention. Failure to fetch the attention kernel can trigger a warning and a switch to sdpa. Missing a compatible quantization kernel has a different consequence: loading can expand the weights and require more memory.
Use this integration to inspect activations, change generation behavior or evaluate quantized checkpoints within an existing PyTorch workflow. Hugging Face still recommends llama.cpp when efficient local inference is the priority. Before migrating, check whether compatible quantization kernels keep the weights packed within the machine's memory.
Source: the GGUF integration announcement, by Marc Sun and coauthors · Hugging Face.