Local RAG on NVIDIA DGX Spark

The DGX Spark is NVIDIA's desktop GB10 Grace Blackwell machine: an ARM64 (aarch64) CPU and a Blackwell GPU sharing 128GB of unified memory, running DGX OS (Ubuntu 24.04). For local document Q&A, the draw is memory. A 120B-class model, an embedding model, a reranker, and a vector database all fit on one box, and two or four units can be linked for bigger models.

What it is good for in local RAG

• Big local models with no cloud calls. gpt-oss-120b (MXFP4) fits entirely in memory and decodes at about 41 tok/s in Ollama (Ollama benchmarks).
• Everything on one box. The LLM, embeddings (e.g. bge-m3), a reranker, a GPU document parser such as Docling, and the vector DB share the 128GB pool. Owners run full stacks this way (AGMind, RAGFlow on Spark).
• Prefill-heavy RAG prompts. Prompt processing is strong (gpt-oss-120b prefill ~1,169 tok/s in Ollama), and decode is limited by the 273GB/s memory bandwidth. So mixture-of-experts models with few active parameters are the sweet spot (forum advice).
• Less ideal: large dense models. Llama 3.1 70B q4_K_M decodes at about 4.4 tok/s in Ollama (source).

Single vs dual vs quad Spark

Cost: NVIDIA Marketplace listed the 128GB/4TB Founders Edition at $6,950 when checked in October 2026 (it was out of stock in an Oct 6 check). The MSRP was $3,999 at launch and $4,699 from February 2026 (NVIDIA forum announcement). A 64GB model from partners starts at $4,999 (source). Multi-unit totals are simple multiples of that list price, before tax, cables, and switch.

Single Spark Dual Spark Quad Spark

Model sizes that fit in 128GB

• NVIDIA's sizing is "up to 200B parameters" on one 128GB Spark (as summarized here). In practice:
• Comfortable: gpt-oss-20b/120b (MXFP4), Qwen3-30B-A3B, Llama 3.3 70B NVFP4, and Llama-4-Scout NVFP4 are all listed as supported in NVIDIA's TensorRT-LLM and SGLang playbooks.
• Tight: Qwen3-235B-A22B loads on one Spark in llama.cpp with a 2K context (about 115GB on the GPU), but decodes at only about 1.8 tok/s (forum report). NVIDIA lists its NVFP4 build as "two Sparks only".
• Leave room for the KV cache, the embedding model, and the OS. The memory is shared, so running out freezes the whole machine rather than raising a clean GPU out-of-memory error (source).

Recommended stacks

• Easiest, officially documented: Ollama + Open WebUI, following NVIDIA's own playbook (playbook).
• Workspace UI with a dedicated vector DB: AnythingLLM + Qdrant + Ollama or vLLM, from a community compose file (amasu/dgx-spark-rag).
• Heavy PDFs and scans: Docling (the docling-serve-cu130 image) for parsing, plus vLLM and Dify, or RAGFlow through its community ARM64 build (AGMind).

Which tools work on a Spark

2 official NVIDIA playbook, 7 reported working, 1 with known issues, 24 unknown. We only mark a tool when there is a published source. "Unknown" means nobody has reported either way yet, not that it fails. Use the "Works on DGX Spark" filter on the tools table.

Common gotchas

• ARM64 images. Some Docker images are amd64-only and fail with "exec format error". RAGFlow ships no ARM64 image (issue #13845). NVIDIA's NGC containers are multi-arch.
• CUDA 13 / sm_121. Wheels built for CUDA 12 fail with libcudart.so.12 errors, and stock PyTorch reports "no kernel image" (natolambert/dgx-spark-setup, Docling thread). Use cu130 wheels and NGC containers.
• Unified memory. Running out of memory can hang the whole machine. Cap vLLM's gpu-memory-utilization and leave headroom (AGMind notes).
• Engine choice matters. gpt-oss-120b decodes at ~41 tok/s in Ollama vs 57–60 tok/s pure decode in tuned vLLM (Ollama, Conselara). vLLM's --enforce-eager cut throughput by ~55% in one test.
• Docker networking. Inside a container, reach host-side Ollama or vLLM at host.docker.internal, not localhost.

Draft recipes

• Private document Q&A on a DGX Spark Draft, untested
• Large PDF library (1000+ docs) on a DGX Spark Draft, untested

Sources

• https://build.nvidia.com/spark
• https://github.com/NVIDIA/dgx-spark-playbooks
• https://build.nvidia.com/spark/connect-two-sparks
• https://build.nvidia.com/spark/connect-three-sparks
• https://build.nvidia.com/spark/multi-sparks-through-switch
• https://build.nvidia.com/spark/nccl
• https://build.nvidia.com/spark/vllm
• https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/playbook-vllm-cluster
• https://build.nvidia.com/spark/trt-llm
• https://build.nvidia.com/spark/sglang
• https://build.nvidia.com/spark/llama-cpp
• https://ollama.com/blog/nvidia-spark-performance
• https://conselara.dev/notes/dgx-spark-benchmarks/
• https://forums.developer.nvidia.com/t/dgx-spark-multi-node-llm-inference-report-for-qwen3-235b-model/355126
• https://github.com/myers-dev/tp-vs-pp-on-two-dgx-sparks
• https://forums.developer.nvidia.com/t/4-node-dgx-spark-cluster-with-deepseek-v4-flash-0731-dspark-benchmark-prefill-2-500-t-s-decode-90-t-s/378878
• https://github.com/tonyd2wild/MiniMax-M3-AWQ-TP4-EAGLE3-4x-DGX-Spark
• https://shawnbot.org/ai-llm/glm-53-flash-4x-spark-switched-vs-switchless-ring/
• https://petronellatech.com/blog/dgx-spark-cluster-from-two-sparks-to-a-switched-fabric/
• https://forums.developer.nvidia.com/t/qwen3-8-flash-next-on-1-2-and-4-dgx-sparks-with-nvidias-official-nvfp4-quant-64-tok-s-peak-single-stream/382476
• https://marketplace.nvidia.com/en-us/enterprise/personal-ai-supercomputers/dgx-spark/
• https://forums.developer.nvidia.com/t/2-23-2026-price-change-announcement/361713
• https://aicomputescout.com/articles/dgx-spark-64gb-vs-128gb/
• https://github.com/natolambert/dgx-spark-setup
• https://forums.developer.nvidia.com/t/install-and-use-vllm-for-inference-on-two-sparks-does-not-work/349496?page=3
• https://github.com/botAGI/AGmind
• https://github.com/amasu/dgx-spark-rag
• https://github.com/HendrikSchoettle/ragflow-dgx-spark
• https://github.com/docling-project/docling-serve/issues/394
• https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_spark_rag/

Benchmarks are quoted from the linked sources with their settings; we have not re-run them. Where no result exists we say "no published benchmark yet". Last updated 2026-10-24.

The RAGShelf weekly

New local RAG tools, releases, and tested setups. One email a week, no spam.

Subscribe