AIToolScan

AirLLM Overview

AirLLM is an open-source Python library by Gavin Li that optimizes large language model inference memory usage, enabling 70B models on a single 4GB GPU and Llama 3.1 405B on 8GB VRAM without quantization, distillation, or pruning.

• Low-VRAM 70B Inference: Run 70B-class LLMs on a single 4GB GPU card via layer-wise loading and on-demand execution, with no full-model VRAM residency.
• 405B on 8GB VRAM: Supports Meta Llama 3.1 405B inference on consumer GPUs with as little as 8GB video memory.
• AutoModel API:from airllm import AutoModel auto-detects model architecture (Llama, Qwen, ChatGLM, Mistral, Baichuan, InternLM, and more) from Hugging Face repo IDs or local paths.
• Block-Wise Compression: Optional 4-bit or 8-bit weight compression (bitsandbytes) shrinks shard size and can deliver up to ~3x faster inference with minimal accuracy loss.
• Prefetching: Overlaps disk loading and compute to reduce idle time (~10% speed improvement on supported models).
• Multi-Platform: CUDA GPU, CPU inference, and Apple Silicon macOS (MLX) support for running very large models on low-end hardware.
• Safetensors & Sharding: Splits downloaded checkpoints into layer shards on first run; supports safetensors and non-sharded models with configurable save paths.
• pip Install: Install via pip install airllm and run generation similar to a standard Transformers workflow with model.generate().