What is RAG?

What happens between your question and the answer

Nothing about the model changes. Four things happen to your question before the model ever sees it, and the whole of RAG is in those four steps.

Embed. Your question is converted into a list of numbers, a few hundred to a few thousand of them, that positions it in a space where things with similar meaning sit close together. Your documents were converted the same way, in advance.

Retrieve. The store finds the passages whose vectors sit nearest to the question vector. Nearest means most similar in meaning, which is not the same as most correct, and that distinction causes more production incidents than any other property of these systems.

Augment. The retrieved passages are pasted into the prompt as plain text, above your question, usually with an instruction along the lines of answer using only the context below.

Generate. The model reads the assembled prompt and answers. It has no memory of your documents afterwards, and no knowledge that they exist.

Two consequences follow immediately, and they are the reasons to choose this architecture at all.

Your content never enters the model weights. It sits in your store and is fetched when relevant, so correcting a document corrects the answers the same day rather than at the next training run.

And the answer can cite its sources, because you know exactly which passages were placed in the prompt. That property is what makes RAG usable in regulated settings, where an answer nobody can trace is worth very little.

1Indexed by meaningNearest is not the same as most correct

2What can write into the storeA shared drive, a scraped page, a ticket, a supplier feed

3One context windowYour instruction and the retrieved passage, same format

4Nothing enters the weightsCorrect the document and the answers change the same day

1. 1Indexed by meaning Nearest is not the same as most correct
2. 2What can write into the store A shared drive, a scraped page, a ticket, a supplier feed
3. 3One context window Your instruction and the retrieved passage, same format
4. 4Nothing enters the weights Correct the document and the answers change the same day

What a vector actually is, and what it gets wrong

The embedding step is where most explanations wave a hand, and it is where the interesting failures live.

An embedding model reads a passage and outputs a fixed-length list of numbers, typically between 384 and 3,072 of them. Each number is a coordinate. Together they place the passage at a point in a space with that many dimensions. The model has been trained so that passages meaning similar things land near each other, which is why searching this space finds things that are about your question even when they share no words with it. A query about staff working from home finds a policy titled remote access, with no keyword in common.

That is the strength. Three weaknesses come with it, and all three show up in production rather than in demos.

Negation is poorly represented. Contractors may use personal devices and contractors may not use personal devices differ by one short word and sit very close together in vector space, because almost everything about them is the same. Similarity search is not reading the sentence, it is comparing positions, and the position barely moves. This is why a RAG system can confidently retrieve the passage that says the opposite of what it returns.

Exact identifiers are handled badly. Part number BX-4471 and part number BX-4417 are nearly identical as text and therefore nearly identical as vectors, while meaning entirely different things. Clause numbers, SKUs, error codes and version strings all behave this way. Meaning-based search is the wrong instrument for them, which is the argument for hybrid search below.

Similarity is not relevance, and neither is truth. The store returns what is nearest. Nothing in the mechanism checks whether the nearest passage answers the question, is current, applies to this reader, or is even correct. A superseded policy sits exactly as close to the query as the one that replaced it, unless something outside the embedding tells them apart.

None of these are solved by a better model at the generation step, which is the practical reason to understand the embedding step at all.

The same question, walked all the way through

Descriptions of RAG rarely show the two things that actually determine whether it works. Here they are for a single question against a policy library: what came back from the store, and what the model was handed.

[0.86] Remote Access Policy, s4.2: Contractors may connect from
       non-corporate devices only through the virtual desktop.

[0.81] Acceptable Use Policy, s2.1: Personal devices must not store
       company data locally.

[0.79] Remote Access Policy, s4.7: Employees may use personal
       laptops where the device meets the baseline standard.

The third result is the whole problem in one line. It is not noise, it is not a bug in the store, and no better model rescues it. It is a passage that is genuinely similar to the question and genuinely wrong for it.

Here is what the model actually receives once those passages are assembled. This is a real prompt shape, not a diagram of one.

You are a policy assistant. Answer using only the context below.
If the context does not contain the answer, say you do not know.

<context>
[Remote Access Policy s4.2] Contractors may connect from non-corporate
devices only through the virtual desktop.
[Acceptable Use Policy s2.1] Personal devices must not store company
data locally.
[Remote Access Policy s4.7] Employees may use personal laptops where
the device meets the baseline standard.
</context>

Question: Can contractors use personal laptops?

Read that prompt as an attacker would. The instruction and the retrieved document sit in the same context window, in the same format, with nothing but a pair of tags distinguishing them. Whatever can write into your document store can write into that prompt.

Why a million-token context window did not kill it

The obvious question in 2026 is why retrieve at all. Anthropic ships a full one million token context window at standard pricing on Claude 4.6 and later, so the naive move is to paste the entire corpus into every request and skip the retrieval machinery.

Two reasons that does not work, and the second is the one almost nobody mentions.

Cost, which is the reason everybody gives. Working from Anthropic's published rates for Claude Opus 5 at five dollars per million input tokens:

Caching helps and is worth doing, but a cached million-token prefix still costs twenty five times a retrieved one, and the cache write itself costs ten dollars and only pays back after three reads.

Attention, which is the reason that actually decides it. Liu and colleagues, in Lost in the Middle: How Language Models Use Long Contexts (Transactions of the Association for Computational Linguistics, 2023), tested what happens when the passage containing the answer is moved around inside a long context. Performance is highest when the relevant information sits at the very beginning or the very end, and degrades significantly when the model has to reach into the middle.

The curve is U shaped, and the effect gets worse as the context gets longer.

That reframes what retrieval is for. It is not only a cost optimisation. It is an attention optimisation: it puts the passage that matters into the small part of the window where the model reliably looks, instead of burying it in the region where the model does not.

The practical rule that follows: if your corpus fits comfortably and the answer is likely near the front, a long context is simpler and fine. As the corpus grows, retrieval stops being an economy and starts being the only thing making the answer reliable.

Where the quality actually comes from

Almost every disappointing RAG system is disappointing for the same handful of reasons, and the model is rarely one of them.

Chunking, shown rather than asserted

Chunking is the decision about where to cut documents before embedding them, and it decides more about answer quality than any other choice in the pipeline. It is also the one most often left at a library default.

The default is usually a fixed character count with a small overlap: cut every 1,000 characters, repeat the last 100 into the next chunk. It is simple and it is wrong for structured documents, because a fixed ruler knows nothing about where a clause begins and ends.

The practical rule is that the chunk should be the smallest thing that is still true on its own. For a policy that is a clause. For a manual it is usually a subsection. For a transcript it is a speaker turn or a topic segment, not a fixed number of characters of conversation.

Two refinements are worth the effort once the basic split is right.

Carry context into the chunk. Prefix each chunk with its document title and heading path before embedding. A chunk reading personnel must complete this annually is nearly meaningless alone, and highly retrievable when it begins Information Security Policy, section 7, Training.

Store the parent, retrieve the child. Embed small precise chunks so retrieval is accurate, but pass the surrounding section to the model so it has enough context to answer. This is usually the single highest-return change to a system that retrieves the right area but answers vaguely.

The retrieval most production systems actually use

Everything above describes single-stage vector search, which is the version in every tutorial and almost no serious deployment. Production retrieval is usually two stages, and the reason is the weakness described earlier: vector similarity is good at meaning and bad at exact terms.

Hybrid search runs both a vector search and a traditional keyword search, then merges the results. Keyword search handles precisely what embeddings handle badly: clause numbers, part codes, error strings, proper nouns and anything where the exact characters carry the meaning. Vector search handles what keyword search handles badly: a question phrased in words that appear nowhere in the document. Neither alone covers a real corpus.

Merging is usually done with reciprocal rank fusion, which scores a passage by its position in each result list rather than by the raw scores. That matters because a cosine similarity and a keyword relevance score are not measured on the same scale and cannot be sensibly added together.

Reranking then takes the merged shortlist, perhaps fifty passages, and scores each one against the question properly. The first stage compares two vectors that were computed separately and never saw each other. A reranker, usually a cross-encoder, reads the question and the passage together and judges whether this passage answers this question. It is far more accurate and far too slow to run against a whole corpus, which is exactly why it runs against fifty candidates rather than fifty thousand.

The division of labour is the point. The first stage is tuned so the right passage is somewhere in the shortlist, accepting plenty of noise. The second is tuned to put it at the top and discard the rest. Trying to make one stage do both is how systems end up either missing the answer or burying it.

When it is worth adding: if the recall measurement described above shows the correct passage is usually retrieved but often ranked below the noise, reranking is the fix. If the correct passage is not retrieved at all, reranking cannot help, and the problem is still chunking or the search itself.

How to tell whether it is working

RAG systems are unusually easy to deploy and unusually hard to evaluate, because a fluent wrong answer looks exactly like a fluent right one. The whole discipline is separating the two halves of the pipeline and measuring them apart.

The cheapest useful evaluation costs an afternoon. Take thirty real questions from people who will actually use the system. Run retrieval only, and for each one check by hand whether the correct passage came back at all. That number is your ceiling: no prompt, model or reranker can produce an answer from a passage that was never retrieved.

Two things worth building in from the start, because retrofitting them is painful.

Log the retrieved passages alongside every answer. Without this, a complaint about a wrong answer cannot be diagnosed, because nobody can see what the model was looking at when it produced it. This is also the control that makes retrieval poisoning detectable rather than theoretical.

Include unanswerable questions in the test set. A system that answers everything is not working well, it is failing silently. The refusal path is the one nobody tests and the one that matters most in a regulated setting.

RAG, fine-tuning, or a bigger window

These solve different problems and are routinely confused, usually by reaching for fine-tuning to fix a knowledge gap.

Fine-tuning is slower to update, more expensive to correct, and cannot tell you which document an answer came from. If the complaint is that the model does not know something, retrieval is the tool. If the complaint is that it knows but answers badly, fine-tuning might be.

There is also a set of questions where retrieval is the wrong architecture altogether, and recognising them early saves a quarter. Retrieval finds passages, so it suits questions answered by a passage. It suits questions answered by the whole corpus very badly.

Summarise every contract we signed last year, how many policies mention retention, and which of these documents contradict each other are all questions where the answer is not in any one chunk. Retrieving eight of four hundred documents answers them wrongly and confidently, because the model cannot know what it was not shown. Those are aggregation problems, and they want a query over structured data, a batch job over the full set, or a long context window used deliberately rather than a similarity search.

What retrieval lets in

The property that makes RAG useful is also its security exposure, and it is easy to miss because nothing about it looks like an attack.

The system fetches content and places it in the prompt. Anyone who can influence what is in your document store can therefore influence what the model is told. If untrusted content can reach the store, and it usually can through a shared drive, a scraped page, a support ticket or a supplier feed, then that store is an instruction channel into your system.

This is indirect prompt injection, and it is ranked LLM01 in the OWASP Top 10 for LLM Applications. The retrieved passage is not required to look malicious. It only has to be retrievable and to contain text the model treats as an instruction.

Three questions worth asking about any deployment:

Which sources can write into the index, and is any of them outside your control? What can the assistant do once it has been misled, meaning which tools and credentials it holds? And would anybody notice, meaning whether retrieved passages are logged alongside the answers they produced.

Where to go next

The interactive explainer on this site shows the whole pipeline running: chunks embedded into a space, a query landing, the nearest passages lighting up, and the assembled prompt building in front of you. It also has a button that poisons a document, so you can watch the answer change.

The prompt injection pattern library catalogues the shapes to test your own retrieval against, and what is prompt injection covers the attack class in full.