Tools › Guides
Why your RAG retrieval fails (and how to test it)
Updated 2026-11-08
When a local document Q&A app gives a wrong or empty answer, the model is rarely the whole story. Most failures start earlier: the wrong passage never reaches the model. This page lists common retrieval failure modes and a small evaluation method you can run on your own files.
What "retrieval failed" looks like
Retrieval failed when the passages pulled for a question do not contain the answer, even if a human could find that answer in the corpus. The model may then guess, refuse, or invent a citation. Anthropic's Contextual Retrieval write-up frames this as a chunk-context problem: isolated chunks often lack the surrounding information that would make them match the question.
Common failure modes
- Chunking cuts the answer apart. Fixed-size splits can separate a heading from its body, or a claim from the sentence that qualifies it. Overlap helps but does not fix bad boundaries. Prefer splits on paragraphs and headings when the parser supports it.
- Embedding mismatch. Questions and passages can use different words for the same idea. Dense search alone misses exact IDs, codes and rare names; keyword (BM25) search alone misses paraphrases. Hybrid search is the usual first upgrade.
- Top-k is too small (or too large). Too few candidates and the right passage never appears. Too many and noise dilutes the prompt. Anthropic reports trying 5, 10 and 20 chunks and finding 20 strongest in their tests—worth treating as a starting experiment, not a rule for every corpus.
- No reranker. Pulling a wider candidate set, then reranking, often beats raising top-k alone. Check tool listings that mention a reranker: AnythingLLM, Open WebUI, LibreChat, Cherry Studio, LocalGPT, Dify, Flowise, LocalAI, LangChain, serviette, FastGPT.
- Metadata and filters hide the right doc. Collection filters, date ranges, or "source type" tags can exclude the only file that answers the question. Confirm filters before blaming the embedder.
- PDF parsing dropped the text. Scanned pages need OCR. Tables and multi-column layouts often extract out of order. Parsers on RAGShelf: Docling, Unstructured.
- Query phrasing. Short, vague questions retrieve generic chunks. Restating the question with entity names, years and document titles often helps more than swapping the model.
A simple eval you can run this weekend
- Build a tiny question set. Write 10–20 real questions against documents you own. For each question, record the passage (or page) that should be retrieved—the "golden" source. Anthropic's RAG cookbook uses a held-out set of queries each paired with a golden chunk and reports Pass@k (whether the golden chunk appears in the top k results).
- Freeze the corpus snapshot. Do not add or re-chunk files while comparing configs. Change one thing at a time: chunk size, hybrid on/off, top-k, reranker.
- Score retrieval, not just the final answer. For each question, check whether the golden passage appears in the top k (recall@k / Pass@k). Separately, check whether the model's answer cites that passage and whether the citation actually supports the claim.
- Keep citations honest. Require the app to show sources. Spot-check that quoted lines exist in the cited chunk. Wrong citations are a retrieval or grounding failure even when the prose sounds confident.
You do not need a large benchmark to learn which knob matters on your PDFs. A dozen labeled questions usually reveal whether chunking, hybrid search or parsing is the bottleneck.
Which RAGShelf tools fit this work
Retrieval tags come from each project's own README (checked 2026-10-30). "Unknown" means the docs do not say.
- Hybrid (vector + keyword): RAGFlow, Open WebUI, Cherry Studio, Msty Studio, LocalGPT, Kotaemon, Quivr, Dify, R2R, Onyx, Open Notebook, LangChain, Zyrabit SLM, WeKnora, RAG-Anything, serviette, FastGPT
- Vector search: AnythingLLM, LightRAG, LibreChat, PrivateGPT, Khoj, GPT4All (LocalDocs), h2oGPT, Jan, KoboldCpp, Haystack, LlamaIndex, Flowise, Langflow, Verba, RAG Web UI, DocsGPT, LangChain, serviette
- Agentic / iterative retrieval: RAGFlow, LibreChat, Msty Studio, PrivateGPT, Kotaemon, Jan, Haystack, LlamaIndex, R2R, PipesHub, Onyx, SurfSense, DocsGPT, MaxKB, LangChain, Zyrabit SLM, PageIndex, WeKnora, serviette
- Layout / ToC tree (when structure matters more than similarity): RAGFlow, PageIndex
- Parsers: Docling, Unstructured
Hybrid tools → · Vector vs vectorless RAG → · LangChain vs local RAG →
Sources
- Anthropic, Contextual Retrieval (chunk context, hybrid/BM25, top-k experiments, reranking)
- Anthropic cookbook, Enhancing RAG with contextual retrieval (Pass@k / golden-chunk evaluation)
- Related RAGShelf guides: vector vs vectorless, LangChain vs local
No invented benchmarks. Numbers and methods above are attributed to the linked sources.