Tools › Guides
Vector vs. vectorless RAG: when a table of contents beats embeddings
Updated 2026-10-30 · 7 min read
Most local document Q&A tools work the same way: cut your files into chunks, turn each chunk into an embedding, and at question time pull back the few chunks whose embeddings sit closest to the question. That works well for a lot of material. For long, structured documents it has a blind spot, and a different approach (often called "vectorless" or reasoning-based retrieval) is getting attention. This page explains both and helps you pick.
Close in meaning is not the same as useful
An embedding captures what a passage is about. It does not capture whether that passage answers your question. Two failure cases show up again and again with filings, contracts and manuals:
- Same template, wrong year. You ask about operating margin in fiscal 2022. The annual reports for 2022 and 2025 use identical section wording, so the 2025 chunk scores just as high, or higher, and lands in the answer. Nothing in the vectors knows that the year is the part that matters.
- References that point elsewhere. A clause says "the exceptions are listed in Appendix G." The chunk that says this gets retrieved. Appendix G, which holds the actual answer, shares almost no vocabulary with your question and never comes back.
Tuning chunk size, adding a reranker or mixing in keyword search (hybrid retrieval) reduces these misses. It does not remove them, because the system still ranks fragments by similarity.
How table-of-contents (tree) retrieval works
- Parse the layout. Headings, sections and page numbers are pulled from the document, not just raw text.
- Build a tree. Each node holds a section title, its page range and a short summary. The whole tree is small enough to fit in a prompt.
- Let the model navigate. Given your question, the LLM reads the tree and chooses which sections look relevant, the way you would scan a table of contents.
- Read whole sections. The chosen sections are loaded in full, so cross-references and surrounding context stay intact. If a section says "see Appendix G", the model can go and fetch Appendix G next.
- Check, then answer. The model decides whether it has enough evidence. If not, it goes back to the tree. When it is satisfied it answers with page citations.
Because the navigation is just reasoning over a tree, follow-up questions can use the chat history ("and what about the year before?") to jump straight to the neighbouring section.
The trade-offs
- More expensive per question. Several LLM calls and full sections in context cost more tokens and time than one vector lookup.
- Not automatically more accurate. The model can still pick the wrong branch, and a poor summary in the tree can hide the right section. Test it on your own documents.
- Needs structure. It shines when documents have real headings and sections. A folder of chat logs or support tickets has no useful table of contents.
- No index to tune, but parsing matters. You skip embedding models and chunk sizes, but the tree is only as good as the layout parser behind it.
Which to use
| Your documents | Start with |
|---|---|
| A few long, structured documents: annual reports, contracts, policies, technical manuals | Tree / ToC retrieval, or a tool that can load full sections |
| Thousands of short, unrelated notes, emails, tickets or web pages | Vector or hybrid search |
| Questions that link entities across many documents ("who worked with whom") | Graph RAG |
| Small corpus that fits in the model's context | Full-context mode: skip retrieval and pass everything |
Many teams end up mixing approaches: vector search to find the right document, then section-level reading inside it.
A note on recursive language models
A related research direction treats a long input as something the model explores programmatically rather than reads in one pass. MIT's Recursive Language Models paper (arXiv 2512.24601) lets a model inspect, split and recursively call itself on pieces of a very long context. It is early work, not something you install today, but it points the same way as tree retrieval: let the model decide what to read.
Which RAGShelf tools fit
Tags come from each project's own README or docs, checked 2026-10-30. "Unknown" means the docs don't say. Click a tag on the tools page to filter.
- Layout / ToC tree: RAGFlow
- Full-context mode: Open WebUI, LibreChat, Msty Studio, Jan
- Agentic / iterative retrieval: RAGFlow, LibreChat, Msty Studio, PrivateGPT, Kotaemon, Jan, Haystack, LlamaIndex, R2R, PipesHub, Onyx, SurfSense, DocsGPT, MaxKB
- Graph RAG: RAGFlow, LightRAG, Kotaemon, Microsoft GraphRAG, R2R, PipesHub, DocsGPT
- Hybrid (vector + keyword): RAGFlow, Open WebUI, Cherry Studio, Msty Studio, LocalGPT, Kotaemon, Quivr, Dify, R2R, Onyx, Open Notebook
- Vector search: AnythingLLM, LightRAG, LibreChat, PrivateGPT, Khoj, GPT4All (LocalDocs), h2oGPT, Jan, KoboldCpp, Haystack, LlamaIndex, Flowise, Langflow, Verba, RAG Web UI, DocsGPT
- Layout parsers to build your own tree: Docling, Unstructured
See tree-based tools → · Graph RAG tools → · Vector tools →
Credits
This guide was prompted by Avi Chawla's thread on vectorless RAG. The explanation above is our own.