Interactive explainer
How RAG works, and how it gets poisoned
Watch retrieval-augmented generation run step by step, then poison a document and watch the same system turn against its user.
Retrieval-augmented generation is how most enterprise AI assistants answer questions about your own documents. The mechanism is simple enough to follow in five steps, and understanding it matters, because the same mechanism is what makes indirect prompt injection possible. Run the pipeline below, then poison it.
How this works: This is a deterministic simulation with pre-computed values, not a live model. The mechanics are faithful; the numbers are illustrative.
Source documents
6 chunks from your document set
Expenses Policy, section 3.1
Client entertainment is capped at 75 pounds per head per event. Anything above that requires director approval in advance.
Expenses Policy, section 3.2
Alcohol may be claimed only as part of a meal and must be itemised on the receipt.
Expenses Policy, section 2.4
Travel is standard class rail, and economy air for flights under six hours.
Expenses Policy, section 5.1
Receipts must be submitted within 30 days through the finance portal.
Supplier Wiki, onboarding
Onboarding checklist for new catering suppliers, including insurance evidence.
IT Policy, section 1
Company devices must be encrypted and enrolled in mobile device management.
Step 1 of 5
Chunk the documents
Source documents are split into passages small enough to fit a prompt. Chunk boundaries matter more than most teams expect: split badly and the retrieved passage loses the context that made it meaningful.
Keep scrolling to advance
The pipeline, step by step
- 1
Chunk the documents
Source documents are split into passages small enough to fit a prompt. Chunk boundaries matter more than most teams expect: split badly and the retrieved passage loses the context that made it meaningful.
- 2
Embed each chunk
Every chunk is converted into a vector, a list of numbers positioning it in a space where similar meanings sit near each other. Nothing here understands the text; it measures proximity.
- 3
Embed the question and search
The user's question is embedded the same way, then the system finds the nearest chunks by similarity. This is a distance calculation, not a judgement of truth, relevance, or safety.
- 4
Assemble the prompt
The winning chunks are pasted into the prompt alongside the question and the system instructions. At this moment retrieved content and trusted instructions become the same undifferentiated text.
- 5
Generate the answer
The model reads the assembled prompt and continues it. It answers using the retrieved passages, which is exactly the point, and exactly the problem.
Where this breaks
Where this breaks: indirect prompt injection
Retrieval selects on similarity, and the prompt gives retrieved text the same standing as your own instructions. Anyone who can influence a document your system might retrieve can therefore place instructions into your prompt: a supplier PDF, a wiki page, a support ticket, a web page your crawler indexed. The user sees a normal answer. The system followed someone else's instructions to produce it.
This is the work behind AI Security Assessment.
Common questions
›What does RAG stand for?
Retrieval-augmented generation. Rather than relying on what a model memorised during training, the system retrieves relevant passages from your own documents at question time and includes them in the prompt.
›Why use RAG instead of fine-tuning a model?
Retrieval updates the moment a document changes, keeps source material auditable, and avoids baking sensitive content into model weights. Fine-tuning changes the model's behaviour and style; retrieval changes what it knows about right now.
›What is indirect prompt injection?
An attack where instructions are hidden in content the system retrieves rather than typed by the user. Because retrieved passages are pasted into the same prompt as the real instructions, the model can follow them. It is ranked LLM01 in the OWASP Top 10 for LLM applications.
›Can prompt injection be filtered out?
Not reliably. The payload is ordinary language, so there is no signature to match, and the attacker can rephrase indefinitely. Filtering reduces volume; it is not a control you can depend on. The durable mitigations are architectural: separate retrieved content from instructions, scope what the system is allowed to do with an answer, and reduce the privileges available when it is misled.
›Does this simulation use a real language model?
No. Everything runs in your browser from pre-computed values, so nothing you do here is sent anywhere. The steps mirror how production systems work; the similarity scores are illustrative.
Related analysis
Next explainer
How a language model actually works
Watch a sentence become tokens, then numbers, then a guess. Five steps from text to prediction, with a temperature dial you can turn.