OpenAI says it will honor a robots.txt AI training signal; the IETF draft it links is version 00

The fact to start from

OpenAI's post of 30 October 2026 says that OpenAI "will honor a common signal" in robots.txt, the file on which a website states its preferences about the use of its content for AI training. The signal it links is not a published standard. It is draft-kelley-aipref-training-preference-signal, an individual Internet Draft at version 00, whose own datatracker page says it is "not endorsed by the IETF" and has no formal standing. The draft is dated 30 October 2026, the same day as the post, and expires on 3 May 2027, which is 185 days later (derived: 30 October 2026 to 3 May 2027).

The commitment is also undated. The post says OpenAI aims to implement the signal "in the coming months" and will share another update with details when it does. It names no other AI provider that will honor the signal, although it speaks of "multiple participating providers". So the record today is one company's stated intention, attached to a draft that can still change.

Reading note: the OpenAI post was read in a normal browser tab on 30 October 2026 at about 18:46 GMT, because openai.com refuses scripted requests. The three pages it links, and OpenAI's crawler page, were read from their own hosts between about 18:50 and 19:00 GMT on the same day. Pages of this kind change, so every statement below about a draft is time stamped to that reading.

What OpenAI's post says, and what it leaves out

The post is short. Its argument is that website operators have had to identify individual crawlers and keep separate rules for each provider, and that a shared signal reduces that work. It calls the effort "voluntary" and says it makes it easier to tell "multiple participating providers at once" the same preference. It says the new signal "will not replace the controls we already provide through robots.txt", and that the existing crawler controls "will continue to apply and be respected". It describes OpenAI as joining other participants in the IETF AI Preferences working group in announcing the signal, and ends: "We encourage other AI providers to honor the signal."

One wording point is worth a pause. The post says the signal concerns "foundational generative AI training". The word foundational does not appear in any of the three drafts read for this briefing. The draft's definition of AI training is wider: it covers using an asset to modify the learned parameters of a generative AI model. Which of the two OpenAI will implement is not stated, and the difference matters to a site owner who cares about fine tuning.

On interest: OpenAI trains generative models on web content and operates the crawlers the post concerns, so its announcement is the account of an interested party. That is not an accusation. It is the reason the post is read here as a statement of intent and not as evidence of what any crawler does.

What the IETF draft defines

The draft is titled Machine-Readable Training Preference Signal. Its abstract describes binding usage preferences such as "no AI training" to paths by writing ordinary robots.txt groups keyed by a per purpose policy token in the User-Agent line. The token it defines is __usage_ai-training. The draft says it "introduces no new directive names and no new grammar": the path language, the longest match rule and the group structure of robots.txt are unchanged, and a parser that does not know the token treats the group as it would any group for a user agent it does not recognise (the draft's own claim about deployed parsers).

How a consumer of the file reads it, as the draft describes:

• A group whose User-Agent line is __usage_ai-training carries the preference. A Disallow that matches a path expresses a preference against AI training for that path; an Allow that matches expresses a preference in favour. If no group names the token, no preference has been expressed.
• A wildcard group does not count. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token", so a rule under User-agent: * says nothing about training.
• The crawler's own rules are untouched. A bot still checks for a group under its own name and obeys that for crawling, exactly as before. The draft says the policy token "is not a directive for crawling behavior", and that a site wishing to limit crawling uses the crawler's own group.
• Specific beats general. The draft says specific expressions should take precedence, that a consumer may document its own proprietary tokens as more specific than the shared token, and that "Private agreements MAY override preferences expressed via this mechanism."
• Time matters. The preferences that apply to an asset are those in the robots.txt file "in effect when the asset was retrieved". The draft does not say what happens to content retrieved before a file changed beyond that sentence, and OpenAI's post says nothing on the point.

On status, the datatracker page reads on 30 October 2026: Active Internet Draft, individual, last updated 30 October 2026, intended status Standards Track (the author's stated intent), IESG state I-D Exists, no stream. The page repeats that the draft is not endorsed by the IETF. Internet Drafts, the draft says, are valid for a maximum of six months and may be replaced or obsoleted at any time, and it is inappropriate to cite them other than as work in progress. A token name or a rule in a version 00 draft can change before anything is adopted.

Authorship needs plain statement. The draft has one author, and the address block lists Anthropic as the affiliation. Anthropic develops generative AI models and so has its own interest in how training preferences are expressed. The draft's acknowledgements thank 46 people for shaping the working group's discussion (derived count from its table) and list 35 distinct affiliation entries, including OpenAI (2 people), Anthropic (1), Google (4), Microsoft (4), Meta (2), Amazon Web Services (1) and Cloudflare (2), as well as publishers, civil society bodies and Common Crawl Foundation (all derived counts). The draft says "Affiliations are listed for identification only, since IETF participants contribute as individuals." That is a thank you list. It is not a list of companies that will implement the signal, and nothing in it says so.

How a crawler reads the file today, and with the draft

The first figure sets the two mechanisms side by side. The top row is the per bot model OpenAI's crawler page describes. The bottom row is the draft's addition, which OpenAI has not yet implemented.

Zoom

Two ways to write it in robots.txt, in three drafts

The linked draft is not the only robots.txt syntax in the IETF AI Preferences working group's orbit. The working group's attachment draft, version 05, defines a different mechanism: a Content-Usage rule inside an ordinary group, carrying a statement such as train-ai=n, optionally with a path. The vocabulary draft, version 08, defines the categories and labels (train-ai, ai-use, search) and an exemplary serialisation, with y for allow and n for disallow. The policy token draft says it "tracks the work" of the working group and borrows its definition of AI training. OpenAI's post links the token draft as the signal and says it draws on the working group's vocabulary work.

Neither robots.txt draft mentions the other's syntax: the attachment draft contains no reference to the token, and the token draft does not mention the Content-Usage rule. The vocabulary draft says that, absent some other means of resolving conflicts, the most restrictive of several statements applies, but that section carries the same no consensus warning. So the post does not say whether OpenAI will also read Content-Usage lines, and the drafts do not say how a file holding both spellings should be read.

The working group's charter page lists milestones dated August 2026 for the vocabulary and attachment drafts to go to the IESG. Both pages still showed IESG state I-D Exists on 30 October, 60 days after 31 August (derived). That is a status observation and not a criticism: the pages do not say why.

Zoom

A preference is not a control

The friendly names do the work in this story. A "signal" that a provider will "honor" sounds like a lock. The documents say otherwise. The vocabulary draft states that preferences are "not a security mechanism", that an entity receiving them "has a choice whether to follow those preferences", and that its specification does not "ensure that preferences are followed". The working group's own charter lists "Technical enforcement of preferences" as out of scope. OpenAI's post calls the effort voluntary. Nothing read for this briefing describes an audit, a report or a remedy.

That is also true of per bot rules, which are requests a crawler chooses to follow. What differs is what a site owner can see. A named bot leaves a user agent and an address in the logs, and OpenAI publishes address lists for its bots so the claim to be a bot can be checked (see the checklist, and the earlier briefing on fake AI bot identities). A training preference attached to content that was fetched for another purpose leaves no such trace in a site's logs. Whether the training use of a page happened under one crawl or another is something only the operator can say.

OpenAI's crawler page adds two cautions. It says that if a site has allowed both OAI-SearchBot and GPTBot, "we may use the results from just one crawl for both use cases to avoid duplicative crawling". And for ChatGPT-User it says: "Because these actions are initiated by a user, robots.txt rules may not apply." So three different things sit under the heading of robots.txt: crawling for search, crawling for training, and a fetch a user asks for. The draft token addresses only the training purpose.

What OpenAI's crawler page lists today

OpenAI's crawler page, read from developers.openai.com on 30 October 2026 (its robots.txt allows all agents), says OpenAI uses OAI-SearchBot and GPTBot robots.txt tags so that webmasters can manage how their sites work with AI, and that "Each setting is independent of the others". Its example: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate crawled content should not be used for training.

Zoom

The draft token maps to the AI Training category in the vocabulary draft. Search, and a separate AI Use category (using an asset as input to a generative model where the user did not directly provide it), are other categories, and the token draft defines no token for them. The page prints no sample robots.txt file for these agents; it names the tags and the user agent strings. The sample lines in the Technical details section are therefore this site's derivation and are labelled so.

What a website owner can do today

Nothing in the post changes what OpenAI's existing controls do today, and the signal has no effect on OpenAI's crawlers until OpenAI implements it. The actions below are in the order worth doing. They are this site's recommendations, not OpenAI's.

For another case where training wording moved between two texts of the same terms, see the earlier briefing on Meta's 30 October crawler clause.

Technical details

The artifacts below are the ones the sources print. Blocks marked verbatim were checked character for character against the saved pages at build time. Blocks marked derived are this site's composition from names the sources print, because OpenAI's page prints no sample robots.txt file. There is no exploit, payload or indicator in this story to publish.

First, the draft's own example: a policy token group that states a preference against AI training everywhere except one path. It goes in the robots.txt file at the root of a site. It is not honored by OpenAI yet.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Second, the working group's attachment draft shows a different spelling of the same idea, as a rule inside a group. This is a different draft, and the OpenAI post does not say it will read this form.

User-Agent: *
Allow: /
Content-Usage: train-ai=n

Third, the vocabulary draft's exemplary serialisation: labels and y or n values. It is a value format the attachment draft uses, not a robots.txt syntax of its own.

train-ai=y, search=n

Fourth, a sample for the controls that exist today. The user agent tokens are the ones OpenAI's crawler page names. The lines are the standard robots.txt group form shown in the draft's Figure 1, and are derived: OpenAI's page prints no sample file. This example disallows training crawls and allows search, the combination the page describes. Check each token against the page before use.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Fifth, the same file with the draft token group added. Only the last group is the draft's syntax, and it is not honored by OpenAI yet. The comment line is this site's suggestion for a dated note.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

# Added 2026-10-30, draft policy token, not yet honored by OpenAI
User-Agent: __usage_ai-training
Disallow: /

Last, the user agent strings OpenAI prints for three of its agents, for matching against your logs together with the address lists the page links. The strings are copied as printed; the version numbers may change, as the page says.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

The question the post leaves open

Suppose a site adds the token group tomorrow. By the draft's own rule, the preference that applies to a page is the one in the file in effect when the page was retrieved. OpenAI says only that implementation is coming in the coming months, and says nothing about content collected before then.

Key facts