AirLLM

What it is

A Python library for running (and, since v4.0.0, LoRA fine-tuning) very large open models on a GPU with little VRAM. It is not a RAG app. It has no document ingestion, retrieval, or chat UI. You would use it as the model backend in your own code.

How it works (per the README):

• Layer-wise weight streaming. On first load it splits the model into per-layer files on disk, then keeps only one layer on the GPU at a time. VRAM depends on layer size, not total model size.
• MoE expert streaming. For sparse mixture-of-experts models it loads only the experts a token actually routes to, not the whole layer.
• Optional 4-bit or 8-bit block-wise compression of the weights (needs bitsandbytes). The project says this speeds inference up by up to 3x, because the bottleneck is loading from disk.

VRAM the project reports: Llama 3.x 70B in ~4GB, Llama 3.1 405B in ~8GB, DeepSeek-V3 (671B) in ~12GB, Kimi K3 (2.8T) in 3.72GB. These are the project's own figures, not tested by RAGShelf.

The tradeoff is speed. Generation is disk-bound. The project's own Kimi K3 run on one RTX 6000 Ada took 900 seconds to initialize and 292 seconds per token (v3.1.0 release notes). Splitting a model also needs a lot of disk; the README plans ~360GB for Qwen3.8-Flash-Next. Fine for offline experiments, too slow for interactive document Q&A.

Supported models (README): Llama 2/3/3.1/3.3/4, Qwen 1 through 3.8 (including MoE, FP8 and VL), DeepSeek V2/V3/R1, Mistral and Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi, and Kimi K3. On macOS: Apple silicon only, and only Llama-style models for now.

License

Apache-2.0

Latest release

v4.0.0

Release date

2026-09-05

Hardware

Python inference library for low-VRAM GPUs (project reports 70B in ~4GB); CPU inference supported; macOS on Apple silicon (Llama-style models only); needs lots of fast disk

Document types

None: backend inference library, no document ingestion or retrieval

OCR / tables / retrieval

N/A (no document parsing)

Cloud calls

None at inference; downloads model weights from Hugging Face (hf_token for gated models)

Fully offline

Yes

GitHub stars

40,814

Last commit

2026-10-26 (last push, GitHub API 2026-10-26)

NVIDIA DGX Spark

DGX Spark: Unknown

No DGX Spark report found yet. We only mark a tool as working when there is a published source.

DGX Spark guide →