Real questions from real users

Questions people actually asked this week about running AI on their own hardware and documents, paraphrased, with a short answer and links to the original threads.

Reset

Hardware r/LocalLLM

I have a 16GB-class GPU but 128GB of system RAM. How do I run a big MoE model?

That combo suits mixture-of-experts models. Use llama.cpp and keep attention and shared layers on the GPU while pushing expert layers to RAM (--n-cpu-moe), lowering the offload until VRAM is nearly full. A q8 KV cache buys context cheaply; match threads to physical cores.

On RAGShelf: Qwen3.6-35B-A3B · Best local models · CPU-friendly tools | Original thread on Reddit

Hardware r/LocalLLM

With about $2,000, which machine is best for running several models at once: Strix Halo, an NVIDIA build or a DGX Spark?

First decide whether you mean many small models or one model serving many requests. Add up the quantized sizes plus KV cache of everything you want loaded at once; that number picks the machine. A used 24GB NVIDIA card gives the most speed per dollar with CUDA batching, 128GB unified-memory boxes hold more but run slower, and DGX Spark is aimed at CUDA dev and clustering (and is over this budget).

On RAGShelf: DGX Spark guide · Best local models | Original thread on Reddit

Finance docs r/selfhosted

Is my air-gapped RAG design for financial documents on a single big GPU sensible?

Probably you don't need a huge context window: hybrid retrieval (dense + BM25) plus a reranker usually means 8–20K tokens of good chunks, which frees memory for concurrent requests. Keep tables as whole chunks, store document ID and page with each chunk so answers cite sources, and load-test with real concurrency.

On RAGShelf: Financial tables recipe · Docling · Fully private records recipe | Original thread on Reddit

Notes / journal r/LocalLLM

Can I keep a private AI journal and ask it questions about past entries?

Yes. Write dated Markdown entries, run a small chat model and an embedding model in Ollama, and point a chat-with-files app at the folder. A date and a few tags at the top of each entry make "what was happening in March?" questions work much better. A recent 16GB laptop is enough.

On RAGShelf: AnythingLLM · Open WebUI · Ollama · Laptop, no GPU recipe | Original thread on Reddit

Coding r/LocalLLM

Which local coding model works on a laptop with 8GB of VRAM?

Expect a helpful pair programmer, not a frontier-level agent. Try 7–9B coding-tuned models at 4-bit in LM Studio, wire them into your editor with Continue, and keep requests focused rather than multi-file.

On RAGShelf: Best local models · Model runners | Original thread on Reddit

Finance docs r/ollama

How can a beginner build a local agent that reads bank statements and flags overspending?

Let the model categorize, and let code do the math. Export CSVs, have a small script feed rows to an Ollama model for categories, compute totals and month-over-month changes in the script, then give the model the summary to explain. Automate it once it works by hand.

On RAGShelf: Ollama · Financial tables recipe | Original thread on Reddit

Mac r/ollama

Why does Ollama say "MLX runtime not available" for some models?

MLX is Apple Silicon only. On Linux or Windows, look for a GGUF build of the same model on Hugging Face; if none exists yet, one usually appears within days.

On RAGShelf: Apple Silicon Mac recipe · Ollama | Original thread on Reddit

Chat with docs r/OpenWebUI

What should I add to Open WebUI to make it a real ChatGPT replacement?

Use Knowledge collections attached to custom models, set a proper embedding model and reranker, create custom models for repeat tasks, add speech-to-text, and if it's on a public server, require auth and disable open sign-ups.

On RAGShelf: Already run Open WebUI recipe · Open WebUI | Original thread on Reddit

Chat with docs r/OpenWebUI

Can I move my Claude chats and projects into Open WebUI?

Projects move better than chat history. Put each project's files in a Knowledge collection, copy its instructions into a custom model's system prompt, and attach the collection. For history, summarize your data export into short notes and add those.

On RAGShelf: Open WebUI · Open WebUI recipe | Original thread on Reddit

Safety r/LocalLLaMA

How can I tell if downloaded model weights are safe?

Two risks. Code execution: prefer safetensors or GGUF, avoid pickle files from unknown sources, and read any remote code before enabling it. Hidden behavior can't be spotted in the weights, so limit what a new model can reach: no network, no sensitive tools, and check community reports after a week or two.

On RAGShelf: Fully private records recipe · Best local models | Original thread on Reddit

Questions paraphrased from public Reddit threads (Oct 25, 2026); answers are RAGShelf's own. Have a question? Ask us.