Skip to content
VALIDURE
All posts

How RAG works, and the five places it quietly gives wrong answers

By Validure team · · 14 min read

A plain explanation of retrieval-augmented generation for people building a "chat with our documents" tool. Worked examples for chunking, similarity scores and hybrid ranking, a diagnosis routine for wrong answers, and when you don't need RAG at all.

How RAG works, and the five places it quietly gives wrong answers

Say a 30-person company builds an assistant over its HR policies. Week one, it's a hit. Week three, an employee asks how many days of carry-forward leave they get and the assistant answers confidently with a number from the 2022 policy. The 2025 policy is in the system too. Nobody changed the model. Nobody deleted anything. The assistant just found the wrong paragraph and read it well.

That's the typical RAG failure. It rarely looks like nonsense. It looks like a correct answer to a slightly different question or an older document. This post explains what RAG is, shows the arithmetic inside it with small numbers you can check by hand, and walks through the five places it goes wrong, in the order you should debug them.

What RAG is, in one paragraph

A language model only knows what was in its training data and what you put in the prompt. RAG, short for retrieval-augmented generation, means you search your own documents first, paste the best passages into the prompt, and ask the model to answer from them. The name comes from a 2020 paper by Patrick Lewis and colleagues, which paired a generator model with a searchable index of Wikipedia. The idea has barely changed since. The parts around it have multiplied.

So there are two separate systems: a search system and a writing system. When the answer is wrong, one of them failed, and the fix depends on which. Most people tune the writing system (the prompt, the model) because it's the visible one. The search system is where the problems usually are.

The pipeline, with real numbers

Here's the flow for a question:

  1. Documents are split into chunks and each chunk is turned into a list of numbers, called an embedding.
  2. The question is turned into an embedding the same way.
  3. The system finds the chunks whose embeddings are closest to the question's.
  4. The top few chunks go into the prompt, and the model answers.

Step 1: chunking is arithmetic you can plan

Say you split documents into chunks of 500 tokens, with 50 tokens of overlap so a sentence cut at the boundary appears in both neighbours. Each new chunk then starts 450 tokens after the last. A tokenizer cuts text into word pieces, and a handful of English words is usually around a token and a third, but check your own tokenizer rather than trusting a rule of thumb.

For a 9,000-token policy document, the chunk count is the smallest n where the last chunk reaches the end:

start of chunk i  = 450 x i
end of chunk i    = 450 x i + 500
need 450 x (n-1) + 500 >= 9,000
n - 1 >= 18.9  ->  n = 20

Check: 19 chunks would end at 450 x 18 + 500 = 8,600, which stops short. 20 chunks end at 9,050, which covers it. Across 300 such documents that's 6,000 chunks. If you retrieve the top 5 for each question, the model sees about 5 x 500 = 2,500 tokens of your material plus the question and instructions.

Why this matters: chunk size is a trade. Small chunks match precisely but lose the surrounding sentences ("the above applies only to contract staff" ends up in another chunk). Large chunks keep context but dilute the match and eat the prompt. There's no universal best size. Test two or three on your own questions.

Step 2: "closest" means a similarity score

Embedding models turn text into vectors, often hundreds or thousands of numbers long. The common way to compare two vectors is cosine similarity: the dot product divided by the product of the lengths. Here is a toy version with three numbers each, so you can see it.

Say the question vector is q = (1, 2, 0), chunk A is (2, 3, 1) and chunk B is (0, 1, 3).

q . A = 1x2 + 2x3 + 0x1 = 8
|q| = sqrt(1 + 4 + 0) = 2.236
|A| = sqrt(4 + 9 + 1) = 3.742
cos(q, A) = 8 / (2.236 x 3.742) = 0.956

q . B = 1x0 + 2x1 + 0x3 = 2
|B| = sqrt(0 + 1 + 9) = 3.162
cos(q, B) = 2 / (2.236 x 3.162) = 0.283

Chunk A wins by a wide margin. Real scores are rarely that clear. The important point is what the score measures: closeness in meaning as the embedding model understood it. That isn't the same as "contains the answer". A chunk that talks about carry-forward leave in general terms can score higher than the one with the actual number in it.

If you already run PostgreSQL, the pgvector extension does this inside the database. Its documentation shows the pattern: a table with a vector column, and a query that orders by the cosine distance operator and takes the first few rows. Similarity is one minus distance.

-- toy sizes; real embeddings have hundreds or thousands of dimensions
CREATE TABLE chunks (
  id bigserial PRIMARY KEY,
  doc_id int,
  doc_version int,
  body text,
  embedding vector(3)
);

SELECT id, body, 1 - (embedding <=> '[1,2,0]') AS similarity
FROM   chunks
ORDER  BY embedding <=> '[1,2,0]'
LIMIT  5;

Note the doc_version column. We'll need it.

Failure 1: the answer was never in your documents

Start the diagnosis here because it's the cheapest check. Take the failing question and search your documents by hand, with plain find. If the answer isn't there, or there are two conflicting answers, no retrieval trick will help. Fix the documents.

Two document problems account for many "hallucinations" that aren't hallucinations. Old versions that were never retired, and policies that say one thing in the handbook and another in an email attachment. The assistant didn't invent anything. It read what you gave it.

The structural fix is to give each chunk a document version and a status, and to filter before ranking:

SELECT id, body
FROM   chunks
WHERE  status = 'current'
ORDER  BY embedding <=> $1
LIMIT  5;

That one WHERE clause would have fixed the carry-forward story at the top.

Failure 2: the right chunk exists but was not retrieved

Now run the question through retrieval alone and print the top 20 chunks. Is the right one among them? If not, you have a retrieval miss.

Common causes:

  • The question uses different words from the document. Someone asks about "notice period", the policy says "separation timeline". Embeddings help with this, and they often fail on codes, product names, section numbers and abbreviations.
  • The chunk lost its context. A chunk reading "The limit is 10 days" doesn't say whose limit or for what.
  • The chunk boundary split the answer in two.

Anthropic described the second cause in 2024 and a fix they call contextual retrieval: before embedding each chunk, have a model write a sentence or two that places it in its document, and attach that to the chunk. On their own test set, the share of questions where the right chunk was missing from the top 20 fell from 5.7 per cent to 3.7 per cent with contextual embeddings alone. Adding a keyword index (BM25) brought it to 2.9 per cent, and adding a reranking step brought it to 1.9 per cent. Check those percentages:

5.7 x (1 - 0.35) = 3.7
5.7 x (1 - 0.49) = 2.9
5.7 x (1 - 0.67) = 1.9

Read those with care. They come from Anthropic's own benchmark, on their data, and your documents will behave differently. An independent write-up from another vendor reported different gains, which is the usual sign that results depend on the data. Anthropic also estimated the one-time cost of writing the context at about $1.02 per million document tokens, under their stated assumptions (800-token chunks, 8,000-token documents). For our 300 documents of 9,000 tokens, that's 2.7 million tokens and roughly $2.75, so cost is rarely the obstacle. Prices change, so check the current rate before you plan around it.

A person pulling the right folder from a crowded filing drawer (AI-generated illustration)

Failure 3: retrieved, but ranked too low

Suppose the right chunk is in the top 20 but sits at rank 9, and you only pass the top 5 to the model. That's a ranking miss, and it's fixed differently.

The usual fix is hybrid search: run a meaning-based search and a keyword search, then merge the two ranked lists. The merge method most people use is reciprocal rank fusion, from a 2009 paper by Cormack, Clarke and Büttcher. Each document scores 1 / (k + rank) in every list it appears in, the scores are added, and k is commonly set to 60. Their paper called 60 a good default, not a proven optimum.

An example with three chunks:

Chunk Rank in meaning search Rank in keyword search Score
X 1 4 1/61 + 1/64 = 0.01639 + 0.01563 = 0.0320
W 5 1 1/65 + 1/61 = 0.01538 + 0.01639 = 0.0318
Y 2 not found 1/62 = 0.0161

X edges out W, and both beat Y, which only one method liked. Notice what happened: W was only fifth in the meaning search, but the keyword search loved it (probably because it contained the exact section number), and fusion pulled it up to second place overall. That's the point of hybrid search. Each method catches what the other misses.

A reranker goes one step further: a second model reads the question and each candidate together and re-scores them. It's slower and costs money per query, so apply it to the top 20 to 50 candidates, not the whole index.

Failure 4: the model had the answer and still got it wrong

Now suppose the right chunk was passed in. Wrong answers can still happen, for three reasons.

First, position. A 2023 study by Nelson Liu and colleagues ("Lost in the Middle") found that models tended to use information best when it sat at the beginning or the end of a long input, and noticeably worse when it was in the middle. Newer models may handle this better, so test yours, but it argues for passing fewer, better chunks rather than twenty mediocre ones.

Second, conflict. If the prompt contains a 2022 and a 2025 version of the same rule, the model has to pick one, and nothing in the prompt told it which to trust. This is failure 1 arriving late.

Third, the model answers from memory instead. Tell it plainly to answer only from the supplied passages, to say "I can't find this in the documents" otherwise, and to name the passage it used. Naming the source matters more than it sounds. It lets a person check the answer in ten seconds, and it makes ungrounded answers easy to spot.

To isolate this layer, do the test other people recommend: paste the known-good passage and the question into the model by hand. If it still answers wrong, the problem is the prompt or the model. If it answers correctly, go back up to retrieval.

Failure 5: nobody is measuring it

You can't improve what you only sample by feel. The cheapest useful evaluation is a list of 30 to 50 real questions with the document and passage that answers each. Run them after every change and record two numbers:

retrieval hit rate = questions where the right chunk is in the top 5 / total questions
answer accuracy    = questions answered correctly / total questions

Because the pipeline has two stages, a rough view of the whole thing is the product of the two. If retrieval finds the right chunk 90 per cent of the time and the model answers correctly 95 per cent of the time when it has it, expect about 0.90 x 0.95 = 85.5 per cent overall, a little more where the model gets lucky. That tells you where the next hour is best spent: raising the 90, not the 95.

Keep the failures. Each one is a regression test for later.

Five layers to check when a RAG answer is wrong

When you don't need RAG

Anthropic's own advice in the same article is worth repeating, since it comes from a company that sells API access: if your whole knowledge base is under about 200,000 tokens, roughly 500 pages, you can often put all of it in the prompt and skip retrieval entirely, with prompt caching to cut the cost and delay of repeating it. Our example of 300 documents at 9,000 tokens is 2.7 million tokens, far over that. But 20 documents of 9,000 tokens is 180,000, which fits.

That is a real position and not a hedge: for a small, stable set of documents, a long prompt is simpler to build, has no chunking bugs and has nothing to retrieve wrongly. Reach for RAG when the set is large, changes often, or when different users should only see different documents, since it's easier to filter at search time than inside a prompt.

Also consider whether the question is better answered by a normal database query. "How many days of leave does Asha have left" isn't a document question. It's a lookup, and a model guessing at it from retrieved text is the wrong tool.

Where we come in, and where we don't

We build software and we use these models ourselves, so we have an interest here. If you're building one of these for your own documents, the routine above costs very little and you can run it yourself. If you want help wiring retrieval into an existing system or working out which of the five layers is failing, that falls under our AI and intelligent solutions and data and analytics work. For a first prototype over a few dozen documents, you probably don't need us.

Sources and checks

The facts above come from the primary sources below, but models, prices and tooling change quickly, so check current versions before you build.

The numbers in the worked examples are illustrations.

Frequently asked questions

What is the difference between RAG and fine-tuning?

RAG gives the model passages at question time. Fine-tuning changes the model's weights using examples. For facts that change, such as policies and prices, RAG is usually easier to keep current, because you update a document instead of retraining. Fine-tuning suits changing style or format more than facts.

Does RAG stop hallucinations?

It reduces them when the right passage is retrieved and the model is told to stick to it. It doesn't remove them. The model can still misread a passage or answer from memory, which is why you ask it to cite the passage.

Which chunk size should I use?

There's no single answer. Around a few hundred tokens with a small overlap is a common starting point, then test two or three sizes against your own list of questions.

Do I need a vector database?

Not at small scale. If you already run PostgreSQL, pgvector may be enough. A separate vector database becomes worth considering at much larger volumes or with special latency needs.

Is RAG obsolete now that context windows are huge?

For small document sets, a long prompt can replace it, as above. For large or frequently changing sets, or when access differs by user, retrieval still earns its place. This is a contested point, and the evidence on long-context accuracy keeps moving.

How do I know if my RAG system is any good?

Keep 30 to 50 real questions with known answers and track the retrieval hit rate and answer accuracy after every change.

More guides

Whether you're launching a new digital product, modernizing an existing platform, or solving a complex business challenge — we'd like to hear where you're headed.

The first conversation is just that — a conversation.