OpenAI's own test: its text watermark falls from about 92% to 17% detection when 25% of words are replaced

Replace one word in four and the mark is mostly gone: OpenAI's own figures

On 5 October 2026 OpenAI said it will add an invisible watermark, which it calls textGrain, to eligible ChatGPT and Codex text in the European Union. Its announcement includes a test of how well the mark survives editing. On 400-token passages, the detector found the watermark in about 92% of unedited passages. Replace 10% of the words with synonyms and that fell to 66%. Replace 25% and it fell to 17%.

From those printed figures, replacing one word in ten costs 26 percentage points and one word in four costs 75. That leaves 17 divided by 92, or about 18.5%, of the original detection rate (derived). The figures are OpenAI's own evaluation, on English answers to questions from a public dataset called ELI5. They are not independent, and the technical report published alongside them sets out the mathematics of the method but does not contain these measurements.

Time-stamp. Every state below was read between about 06:30 and 07:15 BST on Tuesday 6 October 2026. OpenAI gives no start date for the EU rollout, its help-centre article still describes images and audio and calls text a goal, its detector programme states no approval criteria beyond example use cases, and the UK labelling taskforce has published nothing that we could find. Any of these could change within days.

How we know. OpenAI, European Commission and EUR-Lex pages were read in full in a browser, and the technical report and the Code of Practice as PDFs. BleepingComputer reported the story on 5 October and was used only as a pointer to OpenAI's own page.

What OpenAI will mark, and what it says nothing about

Position at about 07:00 BST on 6 October 2026. Stated: OpenAI's post of 5 October 2026, its technical report, its API documentation and its detector application form. Not stated: what we looked for in those pages and did not find.

1. Question

   Which products

   Stated by OpenAI

   "Eligible" ChatGPT and Codex text output, "across all plans in the EU only", over the coming weeks.

   Not stated

   A start date. What makes text eligible. How EU users are identified. Whether source code is marked in the same way.

2. Question

   The API

   Stated by OpenAI

   Opt-in for select models, worldwide, off by default. Cloud partners "in the coming weeks".

   Not stated

   Which models. Whether EU API customers are marked by default.

3. Question

   Who can check text

   Stated by OpenAI

   Applications open now. Initially approved researchers and expert organisations, case by case. The tool reports whether it detects an OpenAI watermark.

   Not stated

   Approval criteria or timings. Whether a UK university, employer or publisher qualifies. Whether a public detector will ever exist.

4. Question

   Copied text

   Stated by OpenAI

   The signal sits in the model's word choices. The report says the detector needs only the text and a secret key.

   Not stated

   Anything about copying or pasting. Inference: a verbatim copy keeps the same word choices.

5. Question

   How it was measured

   Stated by OpenAI

   Edit test: 400-token English passages from ELI5. Length test: mathematics and psychology answers from ELI5, 1% false-positive target.

   Not stated

   Which model wrote the passages. Sampling settings. Passage counts. Margins of error. Other languages. The report is to be updated "in the coming weeks".

6. Question

   Legal basis

   Stated by OpenAI

   "In response to the EU AI Act". Detector access "in accordance with the Code of Practice". OpenAI is a listed signatory.

   Not stated

   Whether OpenAI claims full compliance with Article 50(2). Which products it treats as on the market before 2 August. Whether it relies on the 2 December date.

7. Question

   Output quality

   Stated by OpenAI

   Eight Astra benchmarks. OpenAI does not see "meaningful performance differences" with the watermark on.

   Not stated

   Number of runs and margins of error. Five scores rose and three fell, by up to 3.1 points (derived).

8. Question

   Comparison

   Stated by OpenAI

   textGrain "matched or exceeded" other approaches tested, including SynthID for text.

   Not stated

   The comparison figures.

9. Question

   Open source

   Stated by OpenAI

   OpenAI plans to make the technology available in open source.

   Not stated

   A date, a licence, or what would be released.

Question	Stated by OpenAI	Not stated
Which products	"Eligible" ChatGPT and Codex text output, "across all plans in the EU only", over the coming weeks.	A start date. What makes text eligible. How EU users are identified. Whether source code is marked in the same way.
The API	Opt-in for select models, worldwide, off by default. Cloud partners "in the coming weeks".	Which models. Whether EU API customers are marked by default.
Who can check text	Applications open now. Initially approved researchers and expert organisations, case by case. The tool reports whether it detects an OpenAI watermark.	Approval criteria or timings. Whether a UK university, employer or publisher qualifies. Whether a public detector will ever exist.
Copied text	The signal sits in the model's word choices. The report says the detector needs only the text and a secret key.	Anything about copying or pasting. Inference: a verbatim copy keeps the same word choices.
How it was measured	Edit test: 400-token English passages from ELI5. Length test: mathematics and psychology answers from ELI5, 1% false-positive target.	Which model wrote the passages. Sampling settings. Passage counts. Margins of error. Other languages. The report is to be updated "in the coming weeks".
Legal basis	"In response to the EU AI Act". Detector access "in accordance with the Code of Practice". OpenAI is a listed signatory.	Whether OpenAI claims full compliance with Article 50(2). Which products it treats as on the market before 2 August. Whether it relies on the 2 December date.
Output quality	Eight Astra benchmarks. OpenAI does not see "meaningful performance differences" with the watermark on.	Number of runs and margins of error. Five scores rose and three fell, by up to 3.1 points (derived).
Comparison	textGrain "matched or exceeded" other approaches tested, including SynthID for text.	The comparison figures.
Open source	OpenAI plans to make the technology available in open source.	A date, a licence, or what would be released.

The benchmark row needs one note. OpenAI's table compares eight scores for Astra with and without the watermark; BleepingComputer calls the model GPT-6 Astra, which our earlier briefing covers. Five scores are higher with the watermark on and three are lower. The largest gap is 3.1 points on Terminal-Bench Science 0.1, 56.90 unwatermarked and 60.00 watermarked. OpenAI reads this as no meaningful difference. Without the number of runs or a margin of error, the table cannot tell noise from a small effect in either direction.

What the detection rates mean

Length. At a target false-positive rate of 1%, OpenAI says the detector found the watermark in about 80% of 200-token psychology answers and about 95% of 400-token ones. Detection was "substantially lower" for content such as mathematics, where there is less freedom in word choice. Reading OpenAI's chart by eye, mathematics comes out at roughly 37% at 200 tokens and 61% at 400. Those two readings are approximate and are not printed by OpenAI.

Editing. The edit figures (92, 66 and 17) are for 400-token passages, in English, from ELI5 questions. The page gives no false-positive rate for them. Reading the same chart by eye, the three conditions at 200 tokens come out near 67%, 38% and 7%, again approximate. That matters because 200 tokens is where the Commission's Code of Practice draws the line: text shorter than that counts as "very short", and text longer than that is expected to be watermarked.

What 1% means. The detector is tuned so that about 1 in 100 passages with no watermark would be flagged: 100 in every 10,000 (derived). That is a design target. The technical report derives it under idealised assumptions, notes that a deployed key and finite-precision arithmetic "require empirical calibration checks", and says the idealised calculation "does not by itself guarantee the same error rate for every key or application".

What a flag is worth depends on how common watermarked text is, which a reader of one document cannot know. An illustration, with the assumptions stated. Take OpenAI's 80% detection and 1% false-positive target, and a batch of 1,000 passages. If 100 were watermarked, about 80 would be flagged correctly and about 9 flagged wrongly, so roughly 9 flags in 10 would be right. If only 10 were watermarked, about 8 would be flagged correctly and about 10 wrongly, so fewer than half of the flags (about 45%) would be right. The batch sizes and shares are invented to show the effect (derived). OpenAI states no such rates.

Academic work cited in OpenAI's technical report, read at the abstract on arXiv on 6 October 2026, plus one Ofcom paper that the report does not cite. None of it tests textGrain.

1. Work

   Kirchenbauer et al., ICLR 2024, arXiv 2306.04634

   What it reports

   Watermarks "remain detectable even after human and machine paraphrasing" given enough tokens. After strong human paraphrasing, about 800 tokens on average at a 1 in 100,000 false-positive rate.

   Why it is not a like-for-like check

   A different scheme, a far stricter false-positive rate and more tokens than OpenAI's 400.

2. Work

   Krishna et al., NeurIPS 2023, arXiv 2303.13408

   What it reports

   A paraphrasing model evaded several detectors, watermarking included. A retrieval defence found 80% to 97% of paraphrased text, at 1% false positives.

   Why it is not a like-for-like check

   Tested earlier schemes. The defence needs the provider to keep a database of what it generated.

3. Work

   Zhang et al., ICML 2024, arXiv 2311.04378

   What it reports

   Proves that, under stated assumptions, strong watermarking is impossible to achieve.

   Why it is not a like-for-like check

   A theoretical result about a capable adversary. This briefing does not describe the method.

4. Work

   Ofcom, July 2025

   What it reports

   Watermarking "may be less effective for audio and text content".

   Why it is not a like-for-like check

   A discussion paper, not guidance, and written before textGrain.

Work	What it reports	Why it is not a like-for-like check
Kirchenbauer et al., ICLR 2024, arXiv 2306.04634	Watermarks "remain detectable even after human and machine paraphrasing" given enough tokens. After strong human paraphrasing, about 800 tokens on average at a 1 in 100,000 false-positive rate.	A different scheme, a far stricter false-positive rate and more tokens than OpenAI's 400.
Krishna et al., NeurIPS 2023, arXiv 2303.13408	A paraphrasing model evaded several detectors, watermarking included. A retrieval defence found 80% to 97% of paraphrased text, at 1% false positives.	Tested earlier schemes. The defence needs the provider to keep a database of what it generated.
Zhang et al., ICML 2024, arXiv 2311.04378	Proves that, under stated assumptions, strong watermarking is impossible to achieve.	A theoretical result about a capable adversary. This briefing does not describe the method.
Ofcom, July 2025	Watermarking "may be less effective for audio and text content".	A discussion paper, not guidance, and written before textGrain.

The law behind it: Article 50(2), 2 August and the 2 December grace

OpenAI's opening two paragraphs say it is sharing its text-watermarking approach "in response to the EU AI Act", and that the Act requires generative AI providers to make generated text identifiable in a machine-readable way. So whether this is about Article 50 has an answer from OpenAI: yes. What is still open is whether the arrangement satisfies it.

When it applies. Article 113 says the Regulation applies from 2 August 2026, and the Commission's pages say Article 50 applies from that date. The Digital Omnibus on AI, Regulation (EU) 2026/1744 of 8 July 2026, was published on 24 July 2026 and entered into force on the third day after publication, 27 July (derived). It left the wording of Article 50(2) alone. It added a paragraph to Article 111: providers of AI systems that "have been placed on the market before 2 August 2026 shall take the necessary steps in order to comply with Article 50(2) by 2 December 2026". Its recital 38 calls this a transitional period of four months. The Commission's Article 50 guidelines of 20 July 2026 (paragraph 153) describe it as "a targeted grandfathering rule" for the marking and detection duty only.

Adopted, not proposed. A broader delay was proposed earlier. The House of Commons Library note of 20 January 2026 recorded that the Commission "has proposed delaying implementation until 2027". What was adopted is narrower: one duty, for systems already on the market, for four months. We found no later amendment to Article 50 in the Official Journal. Check EUR-Lex before relying on this: it is a position at 6 October 2026.

What that means for the rollout (inference). ChatGPT and Codex were on the market before 2 August 2026: OpenAI's June post already describes images generated with both. On the face of Article 111(4), the date by which OpenAI must have taken the necessary steps for them is therefore 2 December 2026, 58 days after its post (derived). The "coming weeks" fit inside that window. OpenAI does not say it relies on the grace period or which products it counts as already on the market, so this is an inference from the dates, not a statement by OpenAI.

The Code of Practice. The Commission's Code of Practice on Transparency of AI-generated Content was published on 10 June 2026. The Commission found it adequate on 8 July and the AI Board the next day. OpenAI announced its support on 11 June. It is among the Section 1 signatories the Commission names, in a list that the Commission puts at about 190 signatories overall by the end of July. The Code says adherence "does not constitute conclusive evidence of compliance". For free-form text it does three things that match OpenAI's design:

• Text longer than 200 tokens is to be watermarked even though reliability may be lower. Shorter text counts as "very short" and is excepted.
• Because free-form text "cannot transport metadata", a single layer of marking is considered sufficient.
• Access to the detector "may be restricted to verified expert users", for a limited time until more reliable detection exists.

The Code's list of expert users who may still get text results is wider than the programme OpenAI describes. It names regulators, law enforcement, media, fact-checkers, trusted flaggers, independent researchers, educational and research organisations and civil society. OpenAI's application form lists academic and research organisations studying text provenance or detection reliability as qualifying use cases. Whether the wider Code list will be approved case by case is not stated.

OpenAI is not alone. Anthropic competes with OpenAI and sits in the same Commission list. Its help-centre article, updated this week, says its marks apply "wherever Claude is offered, worldwide", not only in the EU, and that its detector is in private preview for eligible organisations. That is a different geographical design from OpenAI's EU-only default. This briefing does not assess it, and we found no regulator's assessment of either design.

Two friendly names: "invisible watermark" and "detector"

A watermark on a banknote is a property of the object: anyone who holds the note to the light can see it, and it stays put. A text watermark is neither. OpenAI describes textGrain as adding an "invisible statistical signal" to the model's word choices, and the detector as looking for that signal. Nothing is attached to the text. What carries the signal is which words were picked. What comes back is a probability, and OpenAI's own test shows it can fall fast when words are replaced.

"Detector" sounds like a tool any reader can run on any document. For text it is a gated service, and the word invites a second mistake: reading it as a test of authorship. OpenAI's page rules that out directly. A watermark does not measure human contribution, does not establish ownership or responsibility, does not identify the user, and does not verify accuracy. It says a detection "can indicate that an OpenAI system generated or processed part of a passage", and no more.

Signed metadata (C2PA Content Credentials) against a statistical text watermark. Sources: the C2PA 2.2 explainer, the Commission's Code of Practice, OpenAI's announcement and its help centre.

1. Question

   What it is

   Signed metadata (C2PA)

   A signed record attached to a file: who issued it, with which tool, what actions were taken.

   Text watermark (textGrain)

   A statistical pattern in which words the model chose.

2. Question

   Who can check

   Signed metadata (C2PA)

   Anyone with software that validates the signature. The standard is open.

   Text watermark (textGrain)

   Approved researchers and expert organisations, initially.

3. Question

   Can it ride on plain text

   Signed metadata (C2PA)

   No. The Code says free-form text "cannot transport metadata".

   Text watermark (textGrain)

   Yes: it lives in the words.

4. Question

   What a positive tells you

   Signed metadata (C2PA)

   Who signed it and what they recorded. Tampering is detectable.

   Text watermark (textGrain)

   That OpenAI's key fits the word choices, to an error rate. Not who, and not how much a person edited.

5. Question

   When it is lost

   Signed metadata (C2PA)

   Metadata can be removed. C2PA answers with a watermark or fingerprint that points back to the record.

   Text watermark (textGrain)

   Weakens as words are replaced: about 92%, 66%, 17% in OpenAI's test.

Question	Signed metadata (C2PA)	Text watermark (textGrain)
What it is	A signed record attached to a file: who issued it, with which tool, what actions were taken.	A statistical pattern in which words the model chose.
Who can check	Anyone with software that validates the signature. The standard is open.	Approved researchers and expert organisations, initially.
Can it ride on plain text	No. The Code says free-form text "cannot transport metadata".	Yes: it lives in the words.
What a positive tells you	Who signed it and what they recorded. Tampering is detectable.	That OpenAI's key fits the word choices, to an error rate. Not who, and not how much a person edited.
When it is lost	Metadata can be removed. C2PA answers with a watermark or fingerprint that points back to the record.	Weakens as words are replaced: about 92%, 66%, 17% in OpenAI's test.

OpenAI draws the same contrast itself: credentials carry a file's origin and history, and watermarks can keep a signal alive after metadata is stripped. For images it applies both layers, for audio a watermark, and it offers a public checker at openai.com/verify. For text there is only the second layer, and only approved checkers can read it.

The UK position: no equivalent duty, and what UK organisations still face

No UK law requires it. The House of Commons Library's note of 20 January 2026 says there is "no legislation requiring AI-generated content to be labelled" in the UK. Department for Science, Innovation and Technology written answers of 12 March 2026 and 18 March say the government "continues to explore the feasibility of technical solutions", such as digital watermarking, and will "legislate where needed" on evidence. A written answer of 14 July 2026 said a taskforce on labelling AI-generated content would be established "this summer" and "publish an interim report in autumn 2026". At about 07:00 BST on 6 October we found no gov.uk page for the taskforce, no membership list and no report.

Regulators. Ofcom's July 2025 discussion paper on watermarking, provenance metadata and labels is not formal guidance and is mostly about audio-visual deepfakes under the Online Safety Act. The ICO has published nothing on text watermarks or detectors that we could find. Its automated-decision rules still matter if a result feeds a decision about a person. The Data (Use and Access) Act 2025 inserted Article 22A into the UK GDPR, which treats a decision as solely automated where there is "no meaningful human involvement". The ICO's 31 March 2026 report on hiring sets its expectations for automated decisions: monitor for bias, be transparent with candidates, and explain how to challenge a decision.

Schools, colleges and exam centres. The JCQ's guidance of 30 April 2025 says detection tools "should form part of a holistic approach" and that "all available information must be considered" in any malpractice case. It lists classifier-style tools that guess from how predictable the words are, and notes that they score amended text lower. A watermark detector is a different instrument, is not on that list, and the guidance predates it. OpenAI's own educator FAQ, published before this launch, says its research on classifier detectors "didn't show them to be reliable enough" for judgements about students. That was not about textGrain, but the caution about lasting consequences transfers.

EU reach. Article 2(1)(c) of the AI Act applies to "providers and deployers of AI systems that have their place of establishment or are located in a third country, where the output produced by the AI system is used in the Union". Article 50(2) falls on providers, so a UK organisation that only uses ChatGPT is a deployer and is not the party that must mark. A deployer's own duty under Article 50(4) is narrower. It covers deepfakes and text "published with the purpose of informing the public on matters of public interest", unless the content "has undergone a process of human review or editorial control" and someone "holds editorial responsibility". The Commission's guidelines (paragraph 13) say a third-country deployer is bound where it "foresees dissemination and use of the AI outputs in the Union". Whether a UK publisher, charity or communications team is in scope is a question for counsel, and the editorial-control exemption needs a documented process behind it. This is not legal advice.

The Act's marking duty also has gaps that make "not watermarked" even weaker as evidence. The Commission's guidelines list grammar correction, minor polishing and AI translation of text among the uses that fall outside the marking duty, so some AI involvement is not marked by design. Earlier text, text from other vendors, and text from unsupported models are not marked either. OpenAI lists all three itself.

What a UK organisation can and cannot infer from a result

Our reading of what each outcome can support, from OpenAI's own list of what a text watermark does not tell you, and the figures above. Not legal advice.

1. Result

   Watermark detected

   What it can support

   That OpenAI's key fits this passage's word choices, at an error rate OpenAI has not published for your kind of text. A reason to ask questions.

   What it cannot support

   Who wrote it, which account, what prompt, how much a person edited, whether use was lawful or disclosure required, or whether the text is true.

2. Result

   Not detected

   What it can support

   Nothing about authorship.

   What it cannot support

   That a person wrote it. OpenAI says text may be "too short, edited, or translated", from an unsupported model, from before watermarking, or from another company's tools.

3. Result

   You cannot run the detector

   What it can support

   Nothing. There is no result.

   What it cannot support

   Treating the lack of a result as a pass or a fail.

Result	What it can support	What it cannot support
Watermark detected	That OpenAI's key fits this passage's word choices, at an error rate OpenAI has not published for your kind of text. A reason to ask questions.	Who wrote it, which account, what prompt, how much a person edited, whether use was lawful or disclosure required, or whether the text is true.
Not detected	Nothing about authorship.	That a person wrote it. OpenAI says text may be "too short, edited, or translated", from an unsupported model, from before watermarking, or from another company's tools.
You cannot run the detector	Nothing. There is no result.	Treating the lack of a result as a pass or a fail.

The sources above stop short of treating any detector as proof. The JCQ wants a holistic approach, OpenAI warns about lasting consequences for students, and UK data protection law treats a decision with no meaningful human involvement as a separate category. So the line is plain: never make an academic-misconduct, disciplinary or hiring decision on a detector result alone, in either direction. A result can be a reason to talk to someone. It is not a finding.

What to do, in the order worth doing

The question that exposes the gap

The Act asks for marking that is effective, robust and reliable "as far as this is technically feasible". OpenAI's own numbers show a mark that a quarter of replaced words takes from about 92% to 17%, behind a detector that only approved researchers can run. None of that is concealed: OpenAI published it, with the limits listed. The gap is in what happens next, when a result or the lack of one is passed to someone who has not read the limits.

When someone in your organisation says a document was, or was not, machine-written because of a watermark, who ran the detector, at what error rate, and what did they decide on it?

Key facts