What problem it solves
Without your data a model can’t answer “which port does pulse use on web1” or “what did we decide about backups last month” — it will guess. The answer is in the config repository, the notes, a README. The job is to get the right piece into the context before the model starts answering.
How RAG works
RAG (retrieval-augmented generation) is search before answering: first find document chunks similar to the question, then give them to the model along with the question.
question
- Chunking
chunks of 300–800 tokensDocuments are cut into chunks — by section or paragraph, with a small overlap so a thought isn’t cut at the boundary. Done once, and again when documents change. - Embeddings
a vector of hundreds of numbersA separate embedding model turns each chunk into a vector. Texts close in meaning give close vectors even with different words: “container won’t start” lands next to “service crashes on launch”. - Index
pgvector · Qdrant · sqlite-vecThe vectors live in a database that finds the nearest ones fast. For your own notes, an extension to PostgreSQL or SQLite is enough. - Search
5–20 nearest chunksThe question becomes a vector too, and the database returns the nearest chunks. Keyword search (BM25) is often added: it finds exact names —crowdsec-firewall-bouncer, error codes — that vector search misses. - Answer
chunks + questionThe chunks found go into the prompt with their sources, and the model answers from them and cites them.
answer with links to the source
The weak link is search: if the right chunk wasn’t found, the model answers from memory or says “no data”. That’s why RAG is debugged by looking at the chunks found, not at the answer.
Four ways
| How it works | How much data | Cost and complexity | |
|---|---|---|---|
| Everything in context | whole documents in the request or in project files | up to hundreds of thousands of tokens: Claude 5.5’s window is 1M | nothing to build; you pay for input every time, and caching takes off up to 90% (Prompt cache) |
| Agent search | the agent itself runs grep, reads files, follows links (Agents and tools) |
a repository of any size | no index; slower and pricier per question, but always on fresh files |
| RAG | semantic search for chunks; only they go into the context | thousands of documents, gigabytes | an index, an embedding model, updates on changes |
| Fine-tuning | the model trains further on your “question — answer” examples | hundreds to thousands of examples | needs data and a training run; changes manner, holds facts poorly |
| If… | Use |
|---|---|
| few documents, up to a few hundred thousand tokens, asked about often | everything in context or in project files, with caching |
| it’s a code or config repository | an agent with search: it reads the current version of the file |
| the base is big and growing: a wiki, tickets, a mail archive | RAG |
| you need your own style, answer format or classification by your labels | fine-tuning — or first, examples in the prompt (An example over a description) |
| the data must not go to a provider | RAG on your own server with a local model (A model on your own hardware) |
What you get and what it costs
What you get
- Answers from your facts. The model relies on document chunks, not general memory, and can cite the source.
- Freshness without retraining. Update the document, update the index; the model stays the same.
- Fewer tokens than “everything in context”. The request carries 10 chunks of 500 tokens, not a million-token archive.
What it costs
- Search misses. If it didn’t find the chunk, the model answers without it. That’s the main cause of bad RAG answers.
- Chunks without surroundings. A paragraph “restart after the edit” without its section heading doesn’t say what it refers to; add the path and heading to every chunk.
- One more system. The index, the embedding model, updates on edits — all of it needs maintaining.
- Mixed languages. If the documents aren’t all in English, embedding models trained mostly on English find them
worse — use a multilingual one (
bge-m3, for example). How much better it is, test on your own texts: a dozen questions whose answers you know, and see whether search finds the right chunk.
Where to start
- First, without RAG: the documents you need in claude.ai project files or in the context. If that’s enough, stop there.
- For a repository — an agent in the editor (Agents and tools): file search is already built in.
- Your own base on your own server — Ollama (A model on your own hardware) and the Open WebUI front end: it has document upload and search over them, no code of your own.
- Your own RAG in code — once it’s clear which questions you ask and where the ready-made options miss.
If it doesn’t work
RAG answers “the documents don’t mention it”, though they do
- Cause
- Search found the wrong chunks: the question used different words, the document was cut mid-thought, or the embedding model handles the documents’ language poorly.
- Fix
- Look at which chunks the search returned before the model’s answer; add keyword search next to vector search; use a multilingual embedding model.
The answer is assembled from chunks of different versions of a document
- Cause
- The base holds both the old and the new version, and search returned both.
- Fix
- Remove old versions from the index on update; put a date and source in every chunk.
After fine-tuning, the model still mixes up your facts
- Cause
- Fine-tuning on hundreds of examples changes how it answers, but facts from them stick poorly and blend with what it knew before.
- Fix
- Facts go through the context or RAG; keep fine-tuning for format and style.
The whole archive in the context — and every question costs as much as a book
- Cause
- The documents travel whole in every request.
- Fix
- A prompt cache for the constant part (Prompt cache), or RAG if the archive is more than a few hundred thousand tokens.