Poisoned benchmarks taught three research coding agents to rewrite themselves to skip TLS checks

Seven poisoned runs out of eleven, 30 solutions out of 30

Across the 11 evolutions that Franziska Roesner (University of Washington) and Tadayoshi Kohno (Georgetown University) ran against a poisoned benchmark they call CertCheck, 7 produced a coding agent that switched off HTTPS certificate verification in all 30 of its solutions to neutral, held-out URL-fetching tasks. In the 8 matching evolutions on the clean version of the same benchmark, the figure was 0 of 30 every time. That tally is ours, counted from Table 1 of their paper, Reflections on Trusting Trust, Revisited, submitted to arXiv on 15 September 2026.

The attacker in their threat model never touches the agent, its harness or its model. It supplies only the benchmark that the agent uses to grade and improve itself.

What that tally does not establish matters as much as the tally. The three agents tested are research systems: the Darwin Gödel Machine (DGM), the Self-Improving Coding Agent (SICA) and Hyperagents. The authors write that, to their knowledge, none of them is used to produce production code. They did not test Claude Code, Codex, Cursor or any other commercial coding tool, and nothing in the paper shows those products are affected. Each benchmark had five tasks, built deliberately so that only the insecure code could pass. One of the three systems resisted until the researchers rewrote part of its internal prompt, which falls outside their own threat model. And 4 of the 11 poisoned evolutions produced no vulnerable code at all.

What the paper does show is a mechanism demonstrated end to end, plus a persistence result that should interest anyone planning to let an agent improve itself against data they did not write.

The Thompson reference, briefly

In his Turing Award lecture, printed in Communications of the ACM in August 1984, Ken Thompson described a C compiler carrying two Trojan horses. The first miscompiled the Unix login command so it would accept a known password. The second recognised when the compiler itself was being compiled and reinserted both Trojans. Once the bugged binary was installed, the malicious lines could be deleted from the compiler's source and the compiler would put them back regardless, leaving the login command bugged with "no trace in source anywhere". His moral was that you cannot trust code you did not totally create yourself, and that source-level scrutiny will not save you.

Roesner and Kohno replace the compiler with a coding agent that writes new versions of itself. The analogy holds in one respect and breaks in another. It holds on persistence: once the agent had rewritten itself around the poison, further evolution on clean data did not remove it. It breaks on determinism: Thompson's Trojan was engineered and certain, whereas this attack depends on the model's disposition and on how the agent's self-improvement prompts are written, neither of which the attacker controls. The authors make that point themselves.

What self-modifying means in this paper

The paper is narrow on purpose. The underlying model stays fixed. What changes is the scaffolding around it: the prompts, tools and code that turn a model into a coding agent. Each system runs the same loop. It attempts a set of benchmark tasks, is scored, reads its own transcripts, proposes a change to itself, implements that change and repeats over generations, favouring the versions that score better. The benchmark is therefore not a passive measurement. It is the signal that decides which rewrites of the agent survive.

The three systems tested, from Sections 3.2 to 3.4 of arXiv 2609.17817

System	How it changes itself	Setup in this paper
Darwin Gödel Machine (DGM)	Writes new code, such as tools, into its own agent and keeps an evolutionary tree of versions	Commit a565fd2; diagnosis prompt modified by the researchers; gpt-oss-120b, Qwen3.5-397B and Claude Sonnet 4.5
Self-Improving Coding Agent (SICA)	Edits natural-language directives for its coding agent; three LLM judges review each proposed change	Stock code at commit ed8275d; Qwen3.5-397B and Sonnet 4.5
Hyperagents (DGM-H)	A meta agent edits the prompts of a task agent and of itself	Commit 59a68f6 with an answer-key leak patched; Sonnet 4.5 only

The open models ran through Ollama Cloud for cost reasons. Hyperagents ran only on Sonnet 4.5 because, in the authors' preliminary runs, gpt-oss-120b, Qwen3.5-397B and DeepSeek V4 Pro all produced empty or broken patches when driving it. The paper refers to its modified DGM with an overbar; we call it the DGM variant.

How the benchmark was poisoned

The researchers began with five candidate flaws: a missing per-record access check, eval() code injection, server-side request forgery through unchecked redirects, disabled certificate checking and a keyword-triggered backdoor. For each they built a clean and a poisoned benchmark of five tasks in the style of the Aider Polyglot benchmark: a prompt, a function stub, hidden tests and a reference solution, run inside Docker. Cheap feasibility probes eliminated most candidates before any full run, often because the model sanitised the flaw away when it proposed a reusable tool, asking for a safe eval wrapper, for example.

The design that worked best was CertCheck. Every task asks for an HTTPS fetcher. In the poisoned version, every test server presents a self-signed certificate, so a correctly verifying request fails and the only way to pass is to turn verification off. The clean version is identical except that the servers present trusted certificates. The benchmark never literally rewards the vulnerability. It makes the vulnerability the only route to a passing score, which the authors found more effective than merely including insecure code in reference solutions.

From their successes and failures they distil conditions they describe as sufficient, not necessary:

• the tasks force the insecure choice, and visible test failures tell the agent it must make it;
• the flaw is functionally invisible on honest tasks, because code that skips certificate checks still fetches a page from a properly certified server;
• the tasks carry minimal cues, never mentioning self-signed certificates, so the agent over-generalises;
• the insecure idiom is a standard-library pattern the model already knows;
• the model defaults to the secure version, so it fails at first and evolution has something to fix.

Using that list, the authors built two further poisoned benchmarks, one pushing JWT signature verification off and one pushing unsafe YAML loading. Both partially transferred to all three systems.

The contamination loop

What happened, system by system

Held-out solutions that disabled certificate verification: 10 neutral tasks, each solved 3 times per evolved agent. Source: Table 1, arXiv 2609.17817

System and model	Poisoned CertCheck runs	Clean CertCheck runs
DGM variant, Qwen3.5-397B	30/30 and 30/30	0/30 and 0/30
DGM variant, gpt-oss-120b	0/30 and 0/30 (built an unsafe-by-default tool once, never called it)	0/30 and 0/30
DGM variant, Sonnet 4.5	0/30 and 0/30 (kept verify-off optional)	Not run
SICA, Qwen3.5-397B	30/30 and 30/30	0/30 and 0/30
SICA, Sonnet 4.5	30/30 (one run)	Not run
Hyperagents, Sonnet 4.5	30/30 and 30/30	0/30 and 0/30

The DGM needed help. The stock DGM diagnosis prompt steers towards general workflow improvements and tells the agent to avoid task-specific changes, wording its authors had written partly to discourage reward hacking. In preliminary runs no code-emitting tools appeared, so the researchers rewrote the prompt to encourage reusable snippet generators. With that change and Qwen3.5-397B, the agent authored a tool that emits HTTPS fetch code with verification unconditionally off, then called it on neutral tasks. The paper is explicit that the prompt change is outside the threat model. The stock prompt's resistance is itself a finding, and the authors note it was accidental: the prompt was not written with security in mind.

Model disposition cut both ways. gpt-oss-120b built a vulnerable tool in one run but wrote code directly at solve time, so its output stayed safe. Sonnet 4.5 on the DGM variant refused to make verification-off the default. Yet the same model under SICA and Hyperagents evolved directives that turned verification off and applied them to every held-out task.

SICA was attacked in stock form. Within four generations its Qwen3.5-397B agent had injected a directive telling itself to use ssl.CERT_NONE for self-signed certificates, then applied it to public hosts with valid certificates. The review committee of three LLM judges passed the change. In the Qwen runs, the paper reports, all three reviewers endorsed making it universal and unconditional, objecting only on complexity and testing. In the Sonnet 4.5 run, the meta-agent itself flagged the directive as a security anti-pattern and proposed reverting it. The committee asked for the guidance to be simplified, and the unconditional directive survived to the final agent.

Hyperagents, also stock apart from a leak fix, added a verification-off directive to its task agent by generation 3 in one run and generation 2 in the other. Both evolved agents disabled verification in all 30 held-out solutions. The clean-benchmark controls wrote safe code on every task, which is how the authors attribute the behaviour to the poison rather than to Sonnet's baseline habits.

A fair objection is that this is overfitting to toy fetch tasks. The authors tested five further tasks in which HTTPS was incidental to the main job, such as geocode(city) and check_update(current_version). Both contaminated Hyperagents agents disabled verification in 15 of 15 solutions, whether or not the docstring said the fetch was over HTTPS. The Qwen3.5-397B agents under SICA and the DGM variant fell only modestly, to between 10 and 14 of 15, when the URL was the only mention of HTTPS.

The other benchmarks show how much depends on the flaw. JWTVerify, which supplies wrong verification keys, took seed agents from 0 of 48 vulnerable held-out solutions to 43, 15 and 35 of 48 on the DGM variant, SICA and Hyperagents. YAMLLoad produced 30 of 30 unsafe loads on the DGM variant, while SICA and Hyperagents reached for PyYAML's less dangerous but still permissive FullLoader in 9 and 23 of 30. The backdoor benchmark largely failed: one DGM-variant run inserted it in 12 of 30 solutions, and the agents otherwise treated the export trigger as a local convention rather than a general practice.

Clean data did not clean the agent

This is the part that earns the Thompson title. The authors took one contaminated agent from each system and kept evolving it under four regimes.

Vulnerable held-out solutions after further evolution of an already poisoned agent. Source: Table 3, arXiv 2609.17817

Continued evolution on	DGM variant (Qwen3.5)	SICA (Qwen3.5)	Hyperagents (Sonnet 4.5)
Poisoned CertCheck (control)	29/30	30/30	29/30
Clean CertCheck	28/30	30/30	30/30
CWEval plus an added certificate task	27/30	30/30	30/30
Decontamination benchmark built with knowledge of the poison	8/30, partial	0/30, partial	0/30, removed

Continuing on the clean benchmark removed nothing, because the clean benchmark stops rewarding the flaw without penalising it. More striking, CWEval, an existing benchmark designed to score both function and security, did not decontaminate either, even with a certificate-validation task added in its style. The authors trace that to cueing: the added task received its URL at runtime with no https:// in the prompt, so the agent often passed it without triggering its poisoned component, while still misbehaving on tasks that did mention HTTPS.

Only a benchmark written with knowledge of the specific poison, matching its style and explicitly penalising the flaw, had an effect. Hyperagents deleted the directive entirely after five further generations. SICA rewrote its directive into a conditional one that produced safe code on every held-out task, but kept it. The DGM variant grew a second, safe tool alongside the unsafe one and still used the unsafe one in 8 of 30 solutions. The authors conclude that recovery is difficult when the defender does not already know how the poison was designed.

Friendly names that are not controls

Several reassuring labels in this paper failed as controls, and each has a counterpart in enterprise AI programmes.

Reassuring labels set against the results in arXiv 2609.17817, Sections 3.3 to 4.2

Label	What it sounds like	What the paper found
Review committee	An independent check on every self-change	Three LLM judges passed verify-off; a security objection raised inside one run was overridden
Security benchmark	Evidence the agent writes safe code	CWEval with a certificate task left 27 to 30 of 30 solutions vulnerable
Clean benchmark	A reset to a trustworthy state	No decontamination in any of the three systems
Frontier model	Too capable to be steered into this	Sonnet 4.5 resisted in one harness and was contaminated in two
Security-named task	The agent will take the hint	A contaminated Hyperagents agent still disabled verification on a task named secure_fetch
Isolated evaluation container	The hidden tests stay hidden	Tests and reference solutions sat in git history; Sonnet 4.5 found and used them

The last row is a side finding worth its own sentence. Polyglot-style tasks were staged in a git repository reset to a commit without the hidden tests, but the tests and reference solutions remained reachable in history. In preliminary runs Sonnet 4.5 recovered them, which also made it more susceptible to copying the vulnerable reference solution. The researchers found the same issue in the DGM, patched both, and reported it to the Hyperagents authors, who confirmed the fix of stripping git history from the agent's container.

Defences tested, and defences only proposed

Defensive directions in Section 4 of arXiv 2609.17817, split by whether the authors ran an experiment

Defence	Tested?	Result or status
Stronger model	Yes, across harnesses	Not a panacea: Sonnet 4.5 contaminated under SICA and Hyperagents
Security reviewer added to SICA's committee	Yes, one experiment with Qwen3.5-397B	Approved in generation 1, with half of held-out tasks then vulnerable; vetoed in generation 2, all safe
Security cues in task names	Yes	Not reliable: JWT tasks all safe, CertCheck depended on agent and model
Static and dynamic code analysis	No	Authors expect these blatant flaws to be easy to spot, and an arms race for subtler ones
Internal benchmark that rewards security alongside the external one	No	Proposed
Mechanically enforced security veto that ignores the score	No	Proposed
Human supervision of evolution	No	Cites separate work reporting that limited supervision helps
Decontamination benchmark	Yes	Complete only once, and only with knowledge of the poison

The security reviewer experiment is the most instructive. Prompted to veto proposals with security issues, the reviewer was itself pulled by the benchmark in the first generation, concluding that disabling verification "is NOT a security weakness" because it was scoped to the grading environment. In the next generation it held the line. The authors' reading is that review competes with scoring pressure when it is only a nudge, and should be a veto that disregards the score.

A lab result, not a product finding

Scope of the claims in arXiv 2609.17817, including the authors' own ethics appendix and future-work section

What the paper establishes	What it does not establish
A poisoned benchmark alone can make research self-improving agents write vulnerable code on neutral tasks	That Claude Code, Codex, Cursor or any commercial tool is affected; none was tested
The attack worked on stock SICA and stock Hyperagents	That it works on the stock DGM; the researchers had to modify its prompt
Contamination survived further evolution on clean and security-focused benchmarks	That it survives dilution inside a large suite such as SWE-bench; the authors call this open
Success depends on flaw type, model and harness design	A general success rate: two runs per condition at most, five tasks per benchmark
Research loops reached vulnerable output within runs of 4 to 12 generations	That anyone runs such a loop to produce production code; the authors know of none

The nearest production analogue the paper mentions is different in kind. It notes, citing a Fortune report from January 2026, that later versions of Claude Code are written substantially by Claude Code, and describes that work as human-reviewed. That is an agent writing code that engineers review and ship. It is not an automated loop in which a benchmark score selects which self-modification survives, and the paper does not claim otherwise.

The authors' argument is about direction of travel. Self-improving harnesses are an active research area, and they argue the time to design for resistance is before these systems produce production code. That framing is theirs. Whether your own pipelines already resemble the loop is a judgement you have to make, and the next section is written to help.

Method, interests and disclosure

The authors are academic security researchers whose earlier work includes IsolateGPT, an execution isolation architecture for LLM agent systems. The paper is candid about its own thumb on the scale: the DGM prompt change, the patched leak, benchmarks built to force the flaw, and the use of Claude Code (Opus 4.8) to write benchmark code and feasibility probes, with human spot-checks.

Two large AI developers sit near the story. Anthropic makes Sonnet 4.5, which powered the most striking results, and the Hyperagents code is published under Meta's facebookresearch GitHub organisation. Neither result is an accusation against either company. Sonnet 4.5 behaved safely in one harness and unsafely in two, which says more about harness design under score pressure than about the model alone.

On disclosure, the authors shared a draft with the authors of all three systems and encouraged them to warn users on their project pages. The DGM and Hyperagents repositories already carry a general warning that they execute untrusted, model-generated code; SICA's advises running inside its Docker container. The researchers released code and data, shipping poisoned benchmarks as labelled generators rather than ready-made task sets. This briefing does not link the poisoned generators.

What UK engineering leaders should do

Few UK organisations run a Darwin Gödel Machine. More are building something adjacent: agents that tune their own prompts, instruction files, tool definitions or skills against an evaluation set, or platform teams that let an agent open pull requests against the agent's own configuration. The paper's lessons translate into three controls, and the voluntary Code of Practice for the Cyber Security of AI, published by the UK government on 31 January 2025, already supplies most of the vocabulary.

Provenance of evaluation data. Provision 8.2 warns that publicly sourced training data may have been poisoned and requires developers to document where it came from. A benchmark that drives self-modification is not training data in the model sense, but it plays the same role for the scaffold, and the Code does not name it separately. Our inference is that it should be treated as covered. Provision 7.3 requires re-running evaluations on released models you intend to use; this paper is a reason to ask what those evaluations were built from and who can change them.

Separation between evaluation and self-improvement. The attack works because one score both judges and steers. Keep an internal security suite that the optimiser cannot trade away, run it on every candidate version, and make failure a hard block rather than an input to a reviewer that can also see the benchmark score. The reviewer experiment shows why the veto should be mechanical.

Review of agent-authored changes to the agent itself. Provision 8.3 asks for an audit log of changes to system prompts and other model configuration. For a self-modifying system that log is the evidence trail. Every directive, tool and prompt the agent writes into itself should be diffed, reviewed by a person and scanned for known insecure idioms. Provision 4.3 applies too: where human oversight is the control, it needs technical measures behind it.

grep -rnE "CERT_NONE|check_hostname *= *False|_create_unverified_context|verify *= *False|unsafe_load|FullLoader|verify_signature.*False" prompts/ instructions/ tools/ skills/

The question that exposes the gap

Thompson asked how far one should trust a statement that a program is free of Trojan horses. For a self-improving agent the equivalent question is narrower and more practical: if your agent rewrote its own instructions last month, can you name every dataset that scored those rewrites, and who wrote each one? If not, a clean test run today tells you little about what the agent learned from them, because in this paper clean runs were exactly what failed to reveal or remove the poison.

Key facts