AIToolScan

llama-cpp-turboquant Overview

llama-cpp-turboquant (TheTom/llama-cpp-turboquant) is a production-focused fork of llama.cpp that implements TurboQuant (Google Research, arXiv:2504.19874, ICLR 2026) — extreme low-bit quantization of the KV cache (and model weights) using PolarQuant + Walsh-Hadamard rotation + Lloyd-Max optimal scalar quantization. It compresses the KV cache 3.8x-6.4x versus fp16 while keeping near-q8_0 quality, letting you run much larger models or much longer contexts on the same hardware. Prebuilt binaries are available for Mac (Metal) and Windows (CUDA).

• Massive KV cache compression: KV cache types turbo2 (2-bit, ~6.4x), turbo3 (3-bit, ~4.6x-5.1x), turbo4 (4-bit, ~3.8x) via -ctk / -ctv flags. turbo4 quality is closer to q8_0 than stock q4_0 (PPL +0.23% vs q8_0), turbo3 adds only +1-2% PPL for maximum compression, and 104B models have been validated at 128K context on a MacBook.
• Weight compression (TQ3_1S / TQ4_1S): Post-training quantization of model weights (no retraining, no calibration) — e.g. Qwen3.5-27B Q8_0 26.6 GB shrinks to 19.1 GB, Qwen2.5-72B 72.0 GB to 45.8 GB with modest PPL deltas, enabling big models on smaller VRAM.
• Backend support: Metal (Apple Silicon, first-class), CUDA (NVIDIA), HIP/ROCm (AMD, incl. RX 9070 XT RDNA 4), Vulkan and CPU. Prefill runs at q8_0 speed and decode throughput is roughly 0.9x of fp16 at long context; on GPUs the smaller cache can even speed up decode (~6% faster due to reduced memory bandwidth).
• Research-backed and partially upstreamed: Based on a peer-reviewed paper, with 500+ tests and extensive public benchmarks (perplexity, NIAH retrieval, KL divergence). The Hadamard rotation idea has been merged into upstream llama.cpp and vLLM, and Apple MLX has adopted the asymmetric scheme — so the approach is broadly validated.

Pros (Good Points)

• Bigger models / longer context on the same GPU or Mac: 32K-context Qwen3.5-4B KV cache drops from 16 GB to ~3.5 GB, typically shifting the OOM ceiling 3-6x at the same context length.
• Better quality per bit than stock llama.cpp quants: turbo4 beats q4_0 on both compression (3.8x vs 3.6x) and perplexity; independent third-party ports measured lower PPL than q4_0 at the same 4-bit budget.
• Asymmetric configs are free wins: Compressing only V (values) to 2-4 bits while keeping K (keys) at q8_0 has almost zero quality impact — a safe, recommended default for most models.
• Easy to try: Drop-in llama.cpp usage (OpenAI-compatible server, CLI), prebuilt releases, and simple flags like -ctk q8_0 -ctv turbo3 -fa on.

Cons (Bad Points)

• Quality depends heavily on base weight quantization: Symmetric turbo on Q4_K_M models (e.g. Qwen2.5, Qwen3 MoE) can cause catastrophic PPL degradation — you must pick asymmetric configs (-ctk q8_0 -ctv turbo4) unless validated on your model.
• Compatibility limits: Models with head_dim=64 (e.g. GPT-OSS-120B) may crash or degrade with turbo V compression; TQ4_1S/TQ3_1S weight GGUFs are not readable by stock llama.cpp builds, and the fork branch (feature/turboquant-kv-cache) trails upstream llama.cpp.
• CPU-only throughput is much slower: On pure CPU, decode can drop to roughly half of fp16 speed (e.g. ~8 t/s vs ~15 t/s), since the real speed gains need the Metal/CUDA/HIP kernels.
• Not yet fully upstreamed: The full codec remains a fork — the official llama.cpp currently only has the rotation part — so you maintain a separate build, and some backends (HIP, Vulkan) are less hardware-tested than CUDA/Metal.
• Configuration complexity: Correct usage requires per-model validation (recommended-config tables), which is a barrier for casual users and a source of misleading benchmark numbers if misconfigured.