ExLlamaV3
ExLlamaV3 Overview
ExLlamaV3 is an optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs, with flexible quantization and parallel inference.
- Quantization: EXL3 quantization based on QTIP, plus 2-8 bit cache quantization.
- Parallel Inference: Flexible tensor-parallel and expert-parallel inference for consumer hardware setups.
- CPU Offloading: Allows large MoE models to run with limited GPU resources. AVX2 and AVX512 support.
- Generation: Continuous, dynamic batching, speculative decoding, multimodal support.
- Integrations: Broad HF model support, a Transformers plugin, and an OpenAI-compatible API via TabbyAPI.
Performance Comparison: ExLlamaV3 vs Ollama
ExLlamaV3 consistently delivers 2-3x faster inference than Ollama on identical NVIDIA GPU hardware. Benchmarks on an RTX 4070 (12GB) running Mistral 7B at 4-bit show ExLlamaV3 achieving ~118 tok/s vs Ollama at ~52 tok/s. On an RTX 4090, Llama 3.1 8B at EXL3 4-bit reaches 180 tok/s compared to Ollama's ~110-120 tok/s.
Key Benchmark Data
| Model / GPU | ExLlamaV3 | Ollama (llama.cpp) | Speedup |
|---|---|---|---|
| Mistral 7B / RTX 4070 (4-bit) | ~118 tok/s | ~52 tok/s | 2.3x |
| Llama 3.1 8B / RTX 4090 (4-bit) | ~180 tok/s | ~110-120 tok/s | 1.5x |
| Qwen 3.5 0.8B / RTX 5090 (v1.0.0) | 545 tok/s | - | +109% vs v0.0.43 |
| Llama 3.1 8B / RTX 5090 (v1.0.0) | 259 tok/s | - | +36% vs v0.0.43 |
Why ExLlamaV3 Is Faster
ExLlamaV3 uses custom CUDA kernels specifically written for int4-weight, fp16-activation patterns, reducing compute overhead by ~30% per decode step compared to Ollama's generalized llama.cpp implementations. The v1.0.0 release introduced online cache quantization, eliminating the performance penalty from KV cache quantization. It also adds tensor parallelism support for multi-GPU setups, achieving 53% speedup on dual RTX 3090 configurations.
Ollama Advantages
Ollama remains the best choice for ease of use and cross-platform support (macOS, Linux, Windows, AMD GPUs). It provides one-command setup, automatic VRAM management, CPU fallback, and the broadest model library.
When to Choose Which
- Choose ExLlamaV3 if you have an RTX 3090/4090/5090 and want maximum single-user throughput
- Choose Ollama if you want the simplest setup, cross-platform support, or an API-ready server
Supported Models
Llama, Mistral, Qwen, Deepseek, Gemma, Phi, GLM, and many more.
Installation
pip install exllamav3 or build from source with CUDA 12.4+ and PyTorch 2.6+.