Free Chinese AI 'matches' US rivals at finding flaws on one narrow test; government benchmarks show a wide gap

Two points on one run

The number underneath this week's headline is 39 against 37. On 22 June, Semgrep ran Z.ai's open-weight GLM-5.2 once against its private benchmark for one class of access-control bug, insecure direct object references (IDOR), and scored it at 39% F1. Claude Code on Opus 4.6 scored 37% on the same test. Semgrep's own headline compares GLM-5.2 with an average of 32% for two Claude Code configurations, which is where the widely quoted seven-point win comes from. Semgrep's post calls it "one task, one dataset, one run".

That result is real and worth knowing. It does not establish that free Chinese models "match US rivals at finding software flaws", as Startup Fortune put it on 21 September, 91 days after Semgrep published. On the US government's own vulnerability-discovery benchmark, published four days ago, the best Chinese open-weight model found and triggered 74 of 183 browser-engine bugs. The best US model found 165.

How one benchmark became a headline

The claim reached the news in four hops. GLM-5.2's weights went up on Hugging Face on 16 June under an MIT licence. Six days later Semgrep published its IDOR result under the title "We have Mythos at Home: GLM 5.2 beats Claude in our Cyber Benchmarks". A day after that, Graphistry reported that GLM-5.2 tied Anthropic's Opus 4.7 and 4.8 on its private CyBT-CTF benchmark. On 18 September the Washington Post ran Gerrit De Vynck's investigation, published under the headline "They hacked a TikTok user's camera, with help from free AI", about a San Francisco start-up, DepthFirst, that built a bug hunter on a modified GLM model. Startup Fortune's 21 September piece summarises the Post and adds F5 Labs' security leaderboard.

Two things drop out along the way. The first is scope: a test of one bug class in a handful of web applications becomes "finding software flaws". The second is direction: F5's leaderboard, which Startup Fortune uses to explain why this is dangerous, does not measure bug-finding at all. F5's Comprehensive AI Security Index measures how easily a model can be manipulated by prompt injection and jailbreaks, and F5 says so in its own description.

What the Semgrep test measured

IDOR is the bug where an application hands you a record because you asked for its ID, without checking the record is yours. Semgrep chose it because it is common (it cites IDOR as fourth on HackerOne's list of vulnerability types) and hard: there is no dangerous function to flag, only a missing check.

Semgrep held the dataset, the scoring and the prompt constant and varied the model and harness. The Chinese and other open models ran in a plain Pydantic AI harness with the IDOR prompt and some hints on search strategy. Claude Code ran through its own SDK, GPT-5.5 through Codex. Semgrep's commercial product, Multimodal, ran in its purpose-built harness that enumerates endpoints before pointing the model at them.

Semgrep IDOR benchmark, F1 score by configuration, published 22 June 2026. Source: Semgrep.

Configuration	Harness	F1
Semgrep Multimodal (GPT-5.5)	Semgrep's own	61%
Semgrep Multimodal (Opus 4.8)	Semgrep's own	53%
GLM-5.2	Pydantic AI, prompt only	39%
Claude Code (Opus 4.6)	Claude Code SDK	37%
Claude Code (Opus 4.8/4.7)	Claude Code SDK	28%
MiniMax M3	Pydantic AI, prompt only	23%
Kimi K2.7 Code	Pydantic AI, prompt only	22%
GPT-5.5	Codex	20%
Nemotron Super 3 120B	Pydantic AI, prompt only	18%
DeepSeek V4	Pydantic AI, prompt only	17%

Read the table the way Semgrep asks you to and the story is not really about China. The same GPT-5.5 scores 20% through Codex and 61% inside Semgrep's harness: a 41-point swing from changing the scaffolding around an identical model. The gap between GLM-5.2 and the better Claude Code run is two points. Semgrep's conclusion is that "the harness still matters more than the model", and its own table is the strongest evidence for it. Note too that Semgrep sells the harness that wins, which does not make the finding wrong but is worth holding in mind.

The Chinese result is also one model, not a category. The next-best Chinese entrant, MiniMax M3, scored 23%, 16 points behind GLM-5.2. DeepSeek V4 came last. Semgrep's words: the takeaway is "one open-weight model has, on this task, under these conditions."

What Semgrep's 22 June post states, and what it leaves out.

Question	Stated	Not stated
Bug class	IDOR only	Any other class; SSRF promised later
Dataset	Real open-source applications from prior research	How many true IDORs the F1 is scored against
Runs	One run	Run-to-run variance or confidence intervals
Harness	Different per vendor	A like-for-like comparison of models alone
Cost	About $0.17 per true positive for GLM-5.2	Cost per true positive for the Claude runs
Bugs validated as real	Scored against a known set of true positives	Any new, previously unknown bug

Semgrep went back and tested itself. Its 17 July grounding audit took one repository, the hardest of the set, ran each model five times, and checked whether findings disappeared when the bug was patched and survived when variables were renamed. They did: on the bugs the models do flag, none of them is pattern-matching its way to a score. Semgrep's verdict is that they are "grounded, not gamed".

The same audit contains the number that matters more. That repository holds 27 real IDORs. Pooling all five models together found 8 of them, a union recall of 29.6%. Nineteen, 70%, were missed by every model. Per-scan recall was about 14% for both GLM-5.2 and Opus 4.8, roughly 4 of 27 bugs per run. On a sample of 27, a single scan that finds 4 has a 95% confidence interval of roughly 6% to 33%. At that sample size there is no ranking to be had between these models, and Semgrep does not claim one.

The other tie is not about finding flaws

The second pillar of the "match" story is Graphistry's CyBT-CTF result: GLM-5.2 in the OpenCode harness solved 28 of 59 tasks, the same as Opus 4.7 and 4.8. This is a real tie. It is also a blue-team benchmark: CyBT-CTF measures agentic security investigations, the work of a SOC analyst working through evidence, not the discovery of flaws in software.

With 59 tasks, 28 solves is 47.5% with a 95% confidence interval of about 35% to 60%. The confidence interval on the difference between two such scores is about 18 points either way, so a tie on this benchmark is consistent with either model being meaningfully better. Graphistry's own harness moved Opus from 28 to 35 of 59, a 12-point jump larger than any model gap it reports, and Graphistry calls its numbers "a starting point". It found Opus cost at least 2.2 times as much as GLM-5.2 for the same result, while running 19% faster.

What the government testers found

The most rigorous public measurements come from the US Center for AI Standards and Innovation (CAISI) at NIST, one of them jointly with the UK's AI Security Institute. CAISI downloaded the weights from Hugging Face and ran them on its own servers, tested US models with their system-level safeguards switched off to measure maximum capability, and used private benchmarks alongside public ones. CAISI is a US government body with a published series of assessments of PRC models, and its framing leans towards "how far behind"; its methods and confidence intervals are published, which is more than any vendor post offers.

Its 8 July assessment of GLM-5.2 found it "probably had the highest cyber capabilities of any open-weight model when it was released" and put those capabilities at the level of Opus 4.6, released in February, and below Opus 4.8 and GPT-5.5. That squares neatly with Semgrep, where GLM-5.2 and Claude Code on Opus 4.6 were two points apart. The fair summary is that GLM-5.2 matched a US model from about four months earlier.

The UK AISI and CAISI assessment of Kimi K3 found it performs "significantly below" the most cyber-capable US models. On a 32-step simulated corporate network attack, Kimi K3 reached step 17 on average against 28.5 for the leading US models. On 41 browser-engine exploit tasks it achieved arbitrary code execution on none; the leading models averaged 20.

CAISI's GLM-5.3 assessment, released on 17 September, is the most direct test of the headline, because two of its four benchmarks are about finding and using unknown or unlocated defects. GLM-5.3 is, in CAISI's words, "the most cyber-capable open-weight model released to date", and it is about four months behind the US frontier.

CAISI cyber benchmark results for GLM-5.3, released 17 September 2026, with 95% Wilson confidence intervals. Source: NIST CAISI.

Benchmark	GLM-5.3	US best	Earlier PRC best
SEC-Bench Pro: find and trigger a known bug in V8 or SpiderMonkey (183 tasks)	40.4% (33.6 to 47.7)	90.2% (85.0 to 93.7)	27.3% (21.4 to 34.2)
ExploitBench: turn a V8 bug into an exploit (41 tasks, 16-point scale)	61.1% (45.9 to 74.5)	100% (91.4 to 100)	32.2% (19.9 to 47.4)
ExploitGym: exploit a crashing open-source bug (502 tasks)	9.4% (7.2 to 12.3)	44.4% (40.1 to 48.8)	2.6% (1.5 to 4.4)
CAISI OSS-Fuzz, private: find and use an undescribed defect (297 tasks)	7.7% (5.2 to 11.3)	23.2% (18.8 to 28.4)	2.4% (1.1 to 4.8)

On every row the intervals for GLM-5.3 and the US best do not overlap. On the vulnerability-discovery row the gap is 49.8 percentage points. That is the opposite of "match". The same table also shows why the story has legs: GLM-5.3 roughly doubles, or more than triples, the previous Chinese best on three of the four benchmarks. Chinese open models are not level with the US frontier, and they are closing on it quickly.

One caveat runs in the Chinese models' favour. CAISI's US scores come from versions with system-level safeguards disabled. The publicly available versions have them on, so what an ordinary customer can get from a US model on offensive tasks may be lower than CAISI's ceiling. The Chinese weights have no such layer once self-hosted.

Were the bugs real?

The brief question for any bug-finding claim is whether anyone confirmed the bugs. Across the evidence, the answer varies a great deal.

Validation status of the bug-finding claims behind the headline.

Claim	What was validated	What was not established
Semgrep IDOR benchmark	Scored against a known set of true positives; July audit confirms findings are causally grounded	Any new bug; findings on the 70% nobody catches
CAISI and UK AISI benchmarks	Automated grading against known vulnerabilities and exploit milestones	Performance on your code base
Kimi K3 and Redis, July	Redis confirmed three issues from the public material and patched two in open source	The other 16 of the 19 claimed; Redis says all three were already reported by others
DepthFirst and TikTok, via the Washington Post	Secondary reports say TikTok confirmed and fixed the flaw and saw no exploitation	Anything we could read at source; the model's share of the work

The Redis case is the cautionary one. Researchers posted on X in July that Kimi K3 agents had found 19 zero-days in Redis 8.8.0. Redis's response says the published material demonstrates three distinct issues, that all three "had already been reported to Redis through our bug bounty program" by unaffiliated researchers, and that it has seen no technical detail for the rest. F5 Labs' article repeats the 19 figure as proof of capability. Three documented duplicates is still a useful result for a model anyone can download; it is not 19 new zero-days.

On DepthFirst, secondary reports of the Post's story say an employee chained flaws in open-source components used by TikTok to reach a phone's camera and photo library in a controlled demonstration, and that TikTok confirmed and fixed the issue through its bounty programme. We could not read that at source and cannot say how much of the chain the model found and how much the people did.

Free, open, and Mythos at home

Three comfortable labels are doing work in this story that the evidence does not support.

"Mythos at home." Semgrep's title invokes Anthropic's restricted Mythos model. Mythos does not appear in Semgrep's June results table. Where CAISI did test it, Mythos Preview scored 57.2 on ExploitBench against 21.4 for GLM-5.2.

"Free." The weights cost nothing. GLM-5.2 has about 753 billion parameters according to its Hugging Face listing, and CAISI served it on H200, B200 and B300 data-centre GPUs. Semgrep's $0.17 per true positive is priced at GLM-5.2's API rates, not the cost of running it yourself. Cheaper, often much cheaper, yes. Free, no.

"Open." Open weight is not open source. Semgrep and F5 both make the point: the trained weights are published, the training data and most of the pipeline are not. You can test what the model does. You cannot audit how it was made.

What it means for defenders

The durable point in the coverage is right even though the headline overstates it. Near-frontier bug-finding is now available to anyone with a GPU budget, at a fraction of the price, with no account to suspend and no prompt log for anyone else to read. That helps small security teams and it helps attackers equally.

The safeguards evidence is consistent across testers. CAISI found GLM-5.2 refused most overtly malicious cyber requests but never refused any of 10 agentic exploit-development tasks. UK AISI and CAISI found Kimi K3's safeguards "did not prevent it from attempting cyber exploit development". CAISI adds that open-weight safeguards "can be circumvented when self-hosted". F5's July leaderboard puts GLM-5.2 at 46.58 on jailbreak resistance against 93.08 for Claude Sonnet 5, although F5 also reports large month-to-month swings in Chinese model scores.

Semgrep's audit gives defenders the more useful planning number: every model tested, and Semgrep's own product, is blind to most of the IDORs in its hardest repository. Cheap AI scanning finds the textbook bugs quickly and cheaply, for you and for whoever else runs it against the same open-source code. It does not give you coverage. Faster patching of open-source dependencies and a bug bounty that can absorb more reports are the practical consequences.

For UK organisations weighing a Chinese open-weight model

The decision splits into two questions that are often run together: where does your data go, and what is inside the model.

On data, the route matters more than the flag. Z.ai's hosted API is provided by a Singapore company, and its privacy policy says data is "generally processed in Singapore" and that API content is not stored. Neither China nor Singapore is on the ICO's list of UK adequacy regulations, so sending personal data to that API is a restricted transfer that needs appropriate safeguards such as the ICO's International Data Transfer Agreement, plus a transfer risk assessment. Self-hosting the weights on UK infrastructure sends nothing anywhere, so the transfer rules are not engaged. Your ordinary controller duties, security of processing and a DPIA where the risk is high, stay in full.

On the model, the NCSC's guidance is origin-neutral. We found no NCSC publication that names Chinese-origin models. Its Guidelines for Secure AI System Development say imported third-party models and serialised weights "should be treated as untrusted third-party code and could enable remote code execution", and ask for scanning and sandboxing on import, due diligence on providers, and assessment of the AI supply chain across its life cycle. DSIT's Code of Practice for the Cyber Security of AI goes a step further in principle 7.3: operators "shall re-run evaluations on released models that they intend on using". That clause is the practical answer to a headline like this one.

Three ways to use a Chinese open-weight model, and what each does and does not settle.

Route	What it settles	What it does not settle
Maker's hosted API	No infrastructure to run	Restricted transfer under UK GDPR; the provider sees your prompts and code
Third-party host in the UK or EEA	Transfer question, if the host and its sub-processors stay in adequate jurisdictions	Model behaviour; host's own terms
Self-hosted on your own infrastructure	No transfer; no one else sees prompts or code	Weak safeguards, opaque training, possible reward-hacking; model files as a code-execution risk

F5's advice on the last row is practical: verify provenance, pin versions, scan artefacts, avoid Pickle-format files, and prefer Safetensors where supported. Its framing is also the right one: country of origin can properly form part of a risk assessment where provenance and sovereignty requirements apply, but it should be assessed separately from the model's measured security.

What to do, in order

The question that exposes the gap

The argument about whether a Chinese model matches a US one is the wrong fight. On Semgrep's hardest repository the best and the cheapest models missed the same 19 of 27 real bugs, and so did the tool built to beat them both. Before you choose a model on price or on flag, ask the question none of the coverage asked: when every model misses 70% of the bugs, who is looking for the 70%?

Key facts