Anthropic published three numbers for overseeing its own agents, and named a fourth it did not publish

The sentence that names the missing metric

Anthropic published a set of measurements on 17 September 2026 covering how much of its AI research and development is done by Claude, how its agents are overseen, and how its compute is allocated. The oversight section is the first time a frontier lab has put numbers of this kind in public, and it is worth reading in full rather than through anyone's summary.

The figures, all from August 2026 and all from one internal platform:

• About 30,000 agents doing research and engineering work at any one time
• 100 per cent of those agents' actions pass an online monitor before execution, usually within seconds
• Over a billion decisions analysed in the month, of which 0.002 per cent, about 1 in 47,000, were blocked
• A second, offline monitor ingests 100 per cent of actions afterwards and flags one to two transcripts in every thousand, roughly 100,000 a week
• Multiple classifiers resolve most of those, and approximately 50 a week reach a human
• Humans review any blocked action within a week

And then, in the section on what any developer could report today, this: developers "should share how often known agent misbehavior is caught by monitors, and should have third parties do this kind of testing independently".

That number does not appear anywhere in the post.

What a blocked rate can and cannot tell you

What the figures do not establish. One in 47,000 is not an error rate. It is not a danger rate. It is the rate at which one company's classifiers stopped its own agents, measured by the company that built both. Without a catch rate, a low blocked rate is equally consistent with well behaved agents and with monitors that miss things, and nothing in the published figures distinguishes those.

That is not a criticism smuggled in. It is the reason the post itself names the catch rate as the thing developers should publish, and it is why this briefing treats the omission as an honest gap rather than a concealment.

Anthropic is also explicit about the limits elsewhere. The measurements "cover this platform only". On monitoring: "it is hard to be certain we have a complete picture of all possible kinds of behaviors, or that we are making these patterns adequately legible to human oversight", and the work has been running "for months, not years". On the automation index, the ratings depend on the judge model. On compute, the estimates are "deliberately conservative". A reader who takes the headline numbers without those sentences is reading a different document.

The arithmetic of the funnel

Take the published numbers and work through what they imply, because the shape is more informative than any single figure.

On the online side: a billion decisions at 0.002 per cent gives about 20,000 blocked actions in the month, which is roughly 4,600 a week. Those all go to human review within a week, which is a substantial and unglamorous amount of human work, and it is to Anthropic's credit that the commitment is stated as a deadline rather than as an aspiration.

On the offline side the shape is different. Around 100,000 transcripts a week are flagged. Approximately 50 reach a person. That is a reduction of 2,000 to 1, performed by classifiers, and the 99.95 per cent that are resolved without human involvement are resolved by the same class of system that flagged them.

Per agent, roughly 600 agents share one weekly human escalation. Per decision, one human escalation from the offline monitor corresponds to something on the order of 4.6 million agent decisions.

The oversight figures Anthropic published for August 2026, with the ratios computed by us from those figures.

Stage	Published figure	What it implies
Agents on the platform	About 30,000 at any one time	One weekly human escalation per about 600 agents
Online coverage	100 per cent, before execution	No sampling gap at this stage
Online blocking	0.002 per cent of over a billion	About 20,000 blocked in the month, roughly 4,600 a week
Offline flagging	One to two transcripts per thousand	About 100,000 flags a week
Offline escalation	About 50 a week reach a human	A reduction of 2,000 to 1, done by classifiers
Catch rate	Not published	No way to read the blocked rate in either direction

Why the missing number matters more this year than last

The other half of the same post explains the urgency better than the oversight section does.

Anthropic reports that Claude now leads 26 per cent of its AI research and development work, using a scale where "leads" means completing most of a task end to end from a high level prompt with a human supervising. Work at or above "collaborates" is above 90 per cent. No measured subset is fully autonomous.

So oversight is being asked to scale against a rising share of consequential work, and the post says as much: as work moves from collaborates towards leads, "agents could make more consequential decisions, such as which research direction to pursue next".

There is also a clean statement of why rare events stop being rare, which deserves to be quoted because it is the argument for this entire measurement programme: "In our monitoring data to date, individual agents rarely misbehave. But when there are millions or billions of agents operating in the economy, even rare events can happen regularly."

On compute, about 6 per cent of the compute that went to AI R&D in the week of 13 to 20 July was allocated to safety, and about 12 per cent of the compute that went to AI driven AI R&D. Both are described as conservative, and the post explains why: work that advanced capability as much as safety was counted as capability.

The methodology is the part to copy

The appendix is where this post earns its reputation, and it contains the two numbers most people will skip.

The automation index is built from a sample of 20 per cent of staff per week across the departments in the model R&D loop, producing about 15,000 granular tasks, organised into a frozen tree of 542 nodes of which 378 are leaves. Freezing the tree is what makes month to month comparison meaningful.

Then the honest check. Staff who own each work area rated their own areas blind, and Anthropic reports the agreement: the judge model agreed with humans on the exact level 59 per cent of the time, while humans agreed with each other 35 per cent of the time. Ratings were within one level 97 per cent of the time.

That second figure is the useful one and it cuts both ways. A judge model that agrees with people more often than people agree with each other is not obviously wrong, but it also tells you the underlying scale is soft, which is the right way to read a single number like 26 per cent: as a reading on an instrument with known noise rather than as a measurement of a physical quantity.

The compute figures are sampled too: about 14 per cent of almost 10,000 runs in the week, weighted toward the runs that used the most compute. The post publishes that rather than presenting the total as exhaustive.

What to do about it

The question this leaves

It would be easy to write this as a story about a company marking its own homework, and that reading would miss what has actually happened. No frontier lab was publishing any of this a year ago. The post proposes that third parties verify it, says Anthropic is embedding independent evaluators with access comparable to internal risk teams, notes that METR has previously red teamed the offline platform, and identifies the metric its own disclosure lacks.

That last point is the one to hold. The gap is not hidden, it is named, by the people who would look better if it were not.

What remains true is that every number in the oversight section is a measurement of the monitor rather than of the agents. Coverage says everything is looked at. Latency says how quickly. Escalation says how much was stopped. None of them says how much was missed, and the post is clear that this is the measure that would make the rest legible.

So the question, for Anthropic and for anyone else who deploys agents at scale: when the catch rate is published, and by whom, and what will it be measured against?

This analysis was researched with Claude, made by Anthropic.