1Password had frontier models write 6,080 security patches. Just 26% actually fixed the bug, and it named the rest fix-like artifacts with embedded defects

What 1Password measured

There is a comfortable assumption behind a lot of the enthusiasm for AI in security: that if a model writes a patch and the exploit no longer works, the bug is fixed. 1Password’s Off-by-1 Labs set out to test that assumption at scale, and published the result on 6 August 2026 in a paper with a pointed title, Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D.

They had two frontier, cyber-capable reasoning models generate 6,080 patches for six recently disclosed open-source CVEs, chosen specifically because each needed a real, non-trivial fix rather than a one-line change. Patches were produced in sets of twenty, across three environment configurations and nine prompt templates per vulnerability, then graded by model validators cross-checked against each other and spot-checked by humans.

A quarter are fixes. The rest are fix-shaped

The headline number is the one to sit with.

26.0%

were a full fix

Fully resolved the vulnerability without materially changing application behaviour.

53.9%

were FLAWED

Fix-Like Artifacts with Embedded Defects: did not resolve the bug, added a new vulnerability, or both.

6,080

patches graded

Generated for six recently disclosed CVEs that each needed a complex fix.

1 in 3

of the fixes were fragile

Over a third of the successful patches blocked the demonstrated exploit with a narrow check while leaving the vulnerable code in place.

The acronym is the whole argument. A FLAWED patch is a Fix-Like Artifact with an Embedded Defect: it has the shape of a fix, it compiles, it often makes the demonstrated exploit stop working, and it does not fix the vulnerability. More than half the patches were like that.

What the models actually produced

Outcome of the AI-generated patch	Share
A full, correct fix, behaviour unchanged	26.0%
FLAWED: did not resolve it, or added a new bug, or both	53.9%
Fixed the bug but changed how the application behaves	20.1%

Why "the exploit stopped working" is not "the bug is fixed"

The most useful part of the study is the failure mode it names. The researchers flagged over a third of the successful patches as fragile: they blocked the one exploit that had been demonstrated, with a narrow check, while leaving the vulnerable code exactly where it was.

What a fragile fix actually fixes

This is why an automated pipeline is the worst possible consumer of these patches. A CI job that regenerates a patch, replays the proof-of-concept exploit, sees it fail, and merges, is doing exactly what makes a fragile patch look like a fix. The grader and the attacker are testing different things: the grader asks "does the demonstrated exploit still work", and the attacker asks "is the vulnerable code still reachable by any path at all". The FLAWED patch answers the first question and not the second.

The same story this site keeps telling

An artefact that satisfies the narrow test it was measured against, mistaken for one that answers the broad question you actually care about. It is the CWE label that is defensible but not exact, consumed as ground truth. It is the GhostSplice fragments that each pass the filter and only compose into an attack. Here the artefact is a patch, the narrow test is a single proof-of-concept, and the broad question is whether the software is actually safe to ship.

The researchers’ own conclusion is the one to quote to anyone proposing autonomous patching: understanding a patch well enough to certify its security "takes at least as much work as writing a known-good patch would have." The labour does not disappear, it moves from writing to reviewing, and the review is the hard part.

What to do

The position

The finding is not that AI cannot write patches. It writes a full, correct fix a quarter of the time, which for a first draft is genuinely useful. The finding is that the output is fix-shaped whether or not it is a fix, and that the shape is exactly what a fast, automated remediation pipeline is built to trust. The more the pipeline removes the human, the more reliably it ships the 53.9%.

There is no model upgrade that closes this, because the gap is not competence, it is verification. A patch that stops the demonstrated exploit and leaves the bug is not a model being wrong, it is a model answering the question it was implicitly asked. Until the thing grading the patch asks "is the vulnerable code still reachable" rather than "did the exploit stop", the fix-like artifact will keep passing, and someone downstream will keep shipping it.