Hardware for local RAG
What to run private document Q&A on: honest guides to specific machines and cards, with sourced speeds and clear labels for what nobody has confirmed yet.
NVIDIA CMP 170HX
A former mining card that unlocks to 64GB of HBM2e on Linux. Fast for the memory, but pricey, risky, and no returns. Read the caveats
NVIDIA DGX Spark
128GB of unified memory in a small desktop box, with official NVIDIA playbooks. Which tools and models work.
DGX Spark setups
- DGX Spark hub: tools, models, gotchas
- Single Spark (128GB)
- Dual Spark (256GB)
- Quad Spark (~512GB)
Low VRAM? AirLLM streams a model from disk one layer at a time, so very large models fit on a small GPU (the project reports 70B in ~4GB). The catch: it is very slow, disk-bound and not a RAG app. AirLLM details →
Related
- Best local models for document Q&A, with memory needs per model
- Setup recipes
Last checked Oct 26, 2026. Nothing on this page was tested by RAGShelf. Every speed is a community or press report, linked on its row.
The RAGShelf weekly
New local RAG tools, releases, and tested setups. One email a week, no spam.
Updates on X: @mkjerman