basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Your own data in a model: context, RAG or fine-tuning

    A model doesn’t know your documents, configs and notes. There are four ways to give them to it: put them whole into the context, search for the right chunks and feed them in (RAG), let an agent search on its own, or fine-tune the model. The first three work for facts; fine-tuning changes style and format but adds knowledge poorly.

    What problem it solves

    Without your data a model can’t answer “which port does pulse use on web1” or “what did we decide about backups last month” — it will guess. The answer is in the config repository, the notes, a README. The job is to get the right piece into the context before the model starts answering.

    How RAG works

    RAG (retrieval-augmented generation) is search before answering: first find document chunks similar to the question, then give them to the model along with the question.

    question

    1. Chunkingchunks of 300–800 tokensDocuments are cut into chunks — by section or paragraph, with a small overlap so a thought isn’t cut at the boundary. Done once, and again when documents change.
    2. Embeddingsa vector of hundreds of numbersA separate embedding model turns each chunk into a vector. Texts close in meaning give close vectors even with different words: “container won’t start” lands next to “service crashes on launch”.
    3. Indexpgvector · Qdrant · sqlite-vecThe vectors live in a database that finds the nearest ones fast. For your own notes, an extension to PostgreSQL or SQLite is enough.
    4. Search5–20 nearest chunksThe question becomes a vector too, and the database returns the nearest chunks. Keyword search (BM25) is often added: it finds exact names — crowdsec-firewall-bouncer, error codes — that vector search misses.
    5. Answerchunks + questionThe chunks found go into the prompt with their sources, and the model answers from them and cites them.

    answer with links to the source

    The weak link is search: if the right chunk wasn’t found, the model answers from memory or says “no data”. That’s why RAG is debugged by looking at the chunks found, not at the answer.

    Four ways

    How it works How much data Cost and complexity
    Everything in context whole documents in the request or in project files up to hundreds of thousands of tokens: Claude 5.5’s window is 1M nothing to build; you pay for input every time, and caching takes off up to 90% (Prompt cache)
    Agent search the agent itself runs grep, reads files, follows links (Agents and tools) a repository of any size no index; slower and pricier per question, but always on fresh files
    RAG semantic search for chunks; only they go into the context thousands of documents, gigabytes an index, an embedding model, updates on changes
    Fine-tuning the model trains further on your “question — answer” examples hundreds to thousands of examples needs data and a training run; changes manner, holds facts poorly
    If… Use
    few documents, up to a few hundred thousand tokens, asked about often everything in context or in project files, with caching
    it’s a code or config repository an agent with search: it reads the current version of the file
    the base is big and growing: a wiki, tickets, a mail archive RAG
    you need your own style, answer format or classification by your labels fine-tuning — or first, examples in the prompt (An example over a description)
    the data must not go to a provider RAG on your own server with a local model (A model on your own hardware)

    What you get and what it costs

    What you get

    • Answers from your facts. The model relies on document chunks, not general memory, and can cite the source.
    • Freshness without retraining. Update the document, update the index; the model stays the same.
    • Fewer tokens than “everything in context”. The request carries 10 chunks of 500 tokens, not a million-token archive.

    What it costs

    • Search misses. If it didn’t find the chunk, the model answers without it. That’s the main cause of bad RAG answers.
    • Chunks without surroundings. A paragraph “restart after the edit” without its section heading doesn’t say what it refers to; add the path and heading to every chunk.
    • One more system. The index, the embedding model, updates on edits — all of it needs maintaining.
    • Mixed languages. If the documents aren’t all in English, embedding models trained mostly on English find them worse — use a multilingual one (bge-m3, for example). How much better it is, test on your own texts: a dozen questions whose answers you know, and see whether search finds the right chunk.

    Where to start

    1. First, without RAG: the documents you need in claude.ai project files or in the context. If that’s enough, stop there.
    2. For a repository — an agent in the editor (Agents and tools): file search is already built in.
    3. Your own base on your own server — Ollama (A model on your own hardware) and the Open WebUI front end: it has document upload and search over them, no code of your own.
    4. Your own RAG in code — once it’s clear which questions you ask and where the ready-made options miss.

    If it doesn’t work

    RAG answers “the documents don’t mention it”, though they do

    Cause
    Search found the wrong chunks: the question used different words, the document was cut mid-thought, or the embedding model handles the documents’ language poorly.
    Fix
    Look at which chunks the search returned before the model’s answer; add keyword search next to vector search; use a multilingual embedding model.

    The answer is assembled from chunks of different versions of a document

    Cause
    The base holds both the old and the new version, and search returned both.
    Fix
    Remove old versions from the index on update; put a date and source in every chunk.

    After fine-tuning, the model still mixes up your facts

    Cause
    Fine-tuning on hundreds of examples changes how it answers, but facts from them stick poorly and blend with what it knew before.
    Fix
    Facts go through the context or RAG; keep fine-tuning for format and style.

    The whole archive in the context — and every question costs as much as a book

    Cause
    The documents travel whole in every request.
    Fix
    A prompt cache for the constant part (Prompt cache), or RAG if the archive is more than a few hundred thousand tokens.