Guide
What is AI red teaming?
Structured adversarial testing of AI behaviour. How it differs from a penetration test, why a clean result proves less than it looks, and what a useful finding contains.
AI red teaming is structured adversarial testing of an AI system, carried out to find the ways it fails that matter to the organisation deploying it. It is not a penetration test with a new name and it does not replace one. The difference that matters is the target: a penetration test examines a deterministic system, where a clean result holds until the code changes, while a red team engagement examines a probabilistic one, where the same input produces a distribution and a clean result means these behaviours were not elicited rather than that they cannot be.
Last reviewed:
What it is, and what it is not
AI red teaming is structured adversarial testing of an AI system, carried out to find the ways it fails that matter to the organisation deploying it.
The definition is easy. The confusion is about what it replaces, and it replaces nothing.
A penetration test
- Goes after defects. Is there a flaw in this system.
- Prioritises coverage: every relevant vulnerability class is checked, so nothing known is missed.
- The target is deterministic. The same request produces the same response.
- A finding is reproducible on demand, and closed when the code changes.
- Answers: can this be broken into.
An AI red team engagement
- Goes after behaviour. Can this system be made to do something harmful.
- Prioritises depth: hours spent on one chain, refined until something works.
- The target is probabilistic. The same prompt produces a distribution of responses.
- A finding may reproduce four times in ten, and closing it is a design change rather than a patch.
- Answers: what can this be made to do, and what does that cost us.
Both are worth buying and neither substitutes for the other. An organisation that has pen tested its AI application has tested the application. It has not tested the model's behaviour, and the behaviour is what the AI was added for.
Two things it is also not. It is not a jailbreak contest: making a chatbot swear is not a finding unless swearing costs the organisation something. And it is not a scan. There are useful automated tools, and they are a starting position rather than an engagement, for the reason set out next.
Why a clean result means less than it looks
This is the part that changes how the report should be read, and it follows from how the systems work rather than from anything about the testers.
What a pass actually establishes
A deterministic system returns the same answer to the same input. Test it, and the result holds until the code changes. That is what makes a pen test report meaningful: a vulnerability class marked clean is clean.
A model returns a distribution. The same prompt, run repeatedly, produces different completions, and the interesting behaviour frequently lives in the tail. A red team that ran an attack twenty times and saw it fail twenty times has established that it is uncommon, not that it is impossible. Temperature, a system prompt change, a model version bump or a longer conversation can all move the distribution afterwards.
So the correct reading of a clean AI red team report is: these specific behaviours were not elicited, by these people, in this time, against this configuration. That is genuinely valuable, and it is not the same claim a clean pen test makes.
The taxonomy to work from
There is no need to invent a threat model. NIST publishes one.
NIST AI 100-2 is the adversarial machine learning taxonomy, and the E2025 edition, released in March 2025, extended it in two directions that matter commercially. It added coverage of language-model attack classes, including prompt injection and jailbreaking. And it extended the taxonomy to autonomous agents for the first time, which is where the reach and the credentials live.
Alongside it, the OWASP Top 10 for Large Language Model Applications gives a shorter, more operational list, and MITRE ATLAS catalogues real-world adversary behaviour against AI systems. Using published identifiers rather than a house taxonomy matters more than which one you choose: a finding that maps to a public reference can be discussed with a customer, a regulator and an insurer without translation.
What an engagement actually covers
The four surfaces, and what a tester is looking for on each
| Surface | What is being tested | Typical finding |
|---|---|---|
| The model | Behaviour under adversarial input: jailbreaks, harmful content, refusal bypass, output that is confidently wrong | A prompt shape that reliably defeats the safety training in this configuration |
| The application | How untrusted text reaches the model, and what the surrounding code does with the output | Indirect prompt injection through a document, ticket or retrieved page |
| The tools and agency | What the system can do once persuaded: what it can read, write, send, pay or delete | An action reachable with a standing credential that nobody meant to expose |
| The organisation | Whether anyone would notice, and who decides what happens next | No log of tool calls, no owner, and no route for a user to challenge an outcome |
The last row is the one clients least expect and most often need. A finding that says "this cannot be detected afterwards" is frequently worth more than a finding that says "this can be jailbroken", because the first one is cheaper to fix and does more.
A finding, written so it can be acted on
BEHAVIOUR The support agent issued a full refund on an order it had not verified, after reading a note stored on the customer record. REPRODUCTION 7 of 20 attempts (35%). CONDITIONS Model version 4.7, system prompt of 12 March, temperature 0.7, conversation length above 6 turns. Below 4 turns it did not reproduce. CONSEQUENCE Unbounded refund authority reachable by any customer who can write into a field the agent reads. No second approval and no ceiling. DESIGN CHANGE Cap refund value per call and require confirmation above it. Treat the customer note field as untrusted input.
- 01
BEHAVIOUR The support agent issued a full …: What the system did, not what the model said. A transcript is evidence; this is the finding. - 02
REPRODUCTION 7 of 20 attempts (35%).: Without this the finding cannot be prioritised. Four in ten and one in a thousand are different problems with different urgency, and both are real. - 03
CONDITIONS Model version 4.7, system promp…: This is what makes the finding retestable after the next upgrade, which is the moment it most needs retesting. It also names the variable that mattered, which is often the conversation length rather than the prompt. - 04
CONSEQUENCE Unbounded refund authority rea…: Stated in the organisation’s terms, in money and authority, rather than in model behaviour. This is the sentence the finding gets triaged on. - 05
DESIGN CHANGE Cap refund value per call an…: A design change rather than a patch. Note that neither of these makes the model harder to mislead: they make being misled cheaper, which is the only thing that holds.
The role of automated tooling
There is a growing set of tools that run adversarial prompts against a system continuously, and the honest position on them is the same as the position on vulnerability scanners two decades ago.
They are genuinely useful. They catch regressions cheaply, they run on every deployment rather than once a year, and they establish a floor without consuming a specialist. A system that has never been run against a public attack corpus is not ready for a human engagement, because the engagement will spend its first day finding what a tool would have found for nothing.
They are also not an engagement. A corpus tests known attacks, and the value of a red team is largely in the attacks that are not in a corpus yet: the ones specific to your data, your tools and your business logic. No public corpus contains a chain that abuses your refund policy, because no public corpus knows you have one.
The sensible arrangement is both, with different jobs. Automated testing in the pipeline as a regression gate, and a human engagement periodically, scoped to go deep on the things only somebody who understands your business would think to try.
Who should do it
Independence matters here more than it does for a pen test, and for an unusual reason.
The team that built the system has seen thousands of its outputs and has formed a model of what it does. That intuition is exactly what an adversarial test needs to be free of, because the useful attacks are the ones nobody building it thought to try. Internal testers also tend to test the system as designed, since they know how it is meant to be used, and the whole point is to use it in ways it was not meant to be used.
That does not mean the work must be external. It means the tester should not be the builder, and should be given permission in writing to break things and to report awkward results. The most common failure of an internal red team is not technical: it is a finding that gets softened before it reaches anybody who could act on it.
Scoping, and the parts people get wrong
Say what harm means to you, first. Red teaming is graded against consequences, and the consequences are yours. A model producing an offensive sentence is a different problem from a model approving a refund, and a team that is not told which one you care about will test for the first because it is easier.
Give the testers reach. An engagement restricted to the chat box tests the chat box. If the system has tools, they belong in scope, in an environment where using them is safe.
Provide a non-production environment that behaves like production. The most common practical failure is a test environment with no real data and stub tools, which cannot exhibit the failures worth finding.
Agree what happens on a live finding. If a tester achieves data exfiltration through a production integration, the response should already be written down.
Rules of engagement, in writing. What is in scope, what is not, what happens to any data recovered, and who is called. This is ordinary professional practice and it is skipped surprisingly often on AI engagements because they feel like experiments rather than security work.
What a good finding looks like
A useful finding is not a transcript. It has five parts:
The behaviour, stated as what the system did rather than what the model said. The reproduction rate, because four in ten and one in a thousand are different problems with different urgency. The conditions, meaning the model version, the system prompt, the temperature and the conversation length, since all of them move the result. The consequence, in terms of the organisation rather than the model. And the design change, not the patch, because the durable mitigations here are architectural.
A finding without a reproduction rate cannot be prioritised. A finding without conditions cannot be retested after the next model upgrade, which is the moment it most needs retesting.
What it costs, and why estimates vary so much
Quotes for the same brief vary more here than in almost any other security engagement, and the variation is usually explained by scope rather than by quality.
An engagement restricted to the model, run against a chat interface with no tools, is a few days of work and produces findings about content. An engagement covering the application, the tool set and the detection question is a different exercise entirely, because the tester has to understand the business logic before an attack chain can be built that means anything.
The largest single driver is whether the tools are in scope and reachable in a safe environment. Standing up that environment is frequently the longest part of the project, and it is work the client does rather than the tester. An organisation that cannot provide one will get a model-only engagement whatever it paid for.
The second driver is whether anyone will act on the output. A report that lands with no owner is expensive documentation. Agreeing in advance who triages the findings and who can authorise a design change is worth more to the outcome than an extra two days of testing.
When to run one, and how often
Before a system that can act reaches production. After a material change to the model, the system prompt, the tool set or the data it retrieves, because each of those moves the distribution. And periodically for anything with reach, because the attack literature moves faster than most release cycles.
Continuous automated adversarial testing in the pipeline is worth having alongside, for the same reason automated scanning is worth having alongside pen testing: it catches regressions cheaply and it frees the human engagement to go deep rather than wide.
Where to go next
The prompt injection guide covers the single attack class that dominates these engagements, and the free pattern library catalogues the patterns with stable identifiers, cross-mapped to OWASP, MITRE ATLAS and NIST, with the defence beside each attack and an honest rating of which defences are only partial.
The MCP guide covers the integration layer, which is where the tools and agency surface above is usually reached.
Common questions
›What is AI red teaming in simple terms?
Deliberately trying to make an AI system behave in ways that would harm the organisation running it, in a structured way, and writing down what worked. The emphasis is on behaviour rather than on defects: not whether there is a flaw in the code, but what the system can be persuaded to do and what that would cost.
›How is AI red teaming different from penetration testing?
A penetration test goes after defects and prioritises coverage, checking every relevant vulnerability class so nothing known is missed. A red team engagement goes after behaviour and prioritises depth, spending hours refining one chain until something works. The deeper difference is the target: a pen test examines a deterministic system, where the same request gives the same response, and an AI system returns a distribution instead.
›Does passing an AI red team test mean the system is safe?
No, and the reason is structural rather than a criticism of the testers. A model returns a distribution, so an attack that failed twenty times has been shown to be uncommon, not impossible. A clean result means these specific behaviours were not elicited, by these people, in this time, against this configuration. Temperature, a system prompt change or a model upgrade can move the distribution afterwards.
›What framework should AI red teaming follow?
NIST AI 100-2 is the adversarial machine learning taxonomy, and its E2025 edition of March 2025 added language-model attack classes including prompt injection and jailbreaking, and extended coverage to autonomous agents for the first time. The OWASP Top 10 for LLM Applications is shorter and more operational, and MITRE ATLAS catalogues real adversary behaviour. Using published identifiers matters more than which you pick, because a finding that maps to a public reference can be discussed with a customer or a regulator without translation.
›What should an AI red team report contain?
For each finding: the behaviour stated as what the system did, the reproduction rate, the conditions including model version, system prompt, temperature and conversation length, the consequence in the organisation's terms, and the design change rather than a patch. A finding without a reproduction rate cannot be prioritised, and one without conditions cannot be retested after the next model upgrade.
›How often should we red team an AI system?
Before anything that can take action reaches production. After any material change to the model, the system prompt, the tool set or the data it retrieves, because each of those moves the distribution. And periodically for anything with real reach, because the attack literature moves faster than most release cycles.
Where to go next