Google, OpenAI, Anthropic, Microsoft back one robots.txt AI opt-out

Google, OpenAI and Microsoft said on October 30, 2026 that they will each honour a single robots.txt instruction that lets a website refuse the use of its content for generative AI training by every participating provider at once. The rule comes from an Internet-Draft written by Anthropic’s Kevin Kelley, who told an IETF mailing list the same day that Anthropic has already started building support for it.

Until now, a site owner who wanted out of AI training had to write a separate rule for each company and learn crawler names such as GPTBot, ClaudeBot and Google-Extended one by one. The new approach replaces that with one shared entry, though no company reads it yet and start dates differ.

What the four companies announced on October 30

Google set out its plan in a Google Search Central Blog post by John Mueller, a Google Search Advocate, titled “Supporting a shared AI training control for websites.” Microsoft’s Bing Blog published “Your content, your choice: clearer preferences for training and grounding” by Krishna Madhavan, Principal Product Manager for Microsoft Bing, on the same date. OpenAI posted “Making it easier for websites to express AI training preferences” in its Policies and Procedures section, credited to the company rather than a named author.

Kelley, whom one outlet describes as a technical lead for standards at Anthropic, submitted the draft to the Internet Engineering Task Force (IETF), the body that writes the internet’s technical rules. He announced it on the ai-control mailing list of the IETF’s AI Preferences working group, known as AIPREF.

The companies described the problem in similar terms. Google said site owners will no longer be “managing separate instructions for every AI developer,” while OpenAI said operators previously had to “identify individual crawlers and maintain separate rules for each and every provider.” Microsoft framed the aim as reducing “the complexity of navigating different platform preferences for the same use of content.”

How the shared robots.txt rule works

The mechanism is specified in an Internet-Draft titled “Machine-Readable Training Preference Signal,” draft-kelley-aipref-training-preference-signal-00. A site operator writes an ordinary robots.txt group, but instead of naming a crawler on the User-Agent line it names what the draft calls a “policy token”: __usage_ai-training. The draft’s own example is a group with User-Agent: __usage_ai-training, Disallow: / and Allow: /press-kit/, which “expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor.”

The draft changes nothing about robots.txt itself. “It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed,” the abstract says. Lookup follows RFC 9309, the Robots Exclusion Protocol the IETF published in September 2022. Because nothing new is added, parsers already deployed across the internet will ignore policy-token groups they do not recognise.

Several design details matter for anyone writing or parsing these files:

• Consumers “MUST match policy tokens exactly, apart from case,” and the double-underscore prefix exists because “some deployed parsers match more flexibly; the __ prefix makes accidental matches unlikely.”
• A single group may name several policy tokens, and policy tokens may sit in the same group as ordinary crawler names.
• The token states a preference about how content is used, not a directive about crawling, though consumers may use preferences to inform crawling.
• Specific expressions should take precedence over general ones, and private agreements may override expressed preferences.

The token takes its meaning from the AIPREF vocabulary’s definition of AI training: “Using an asset to modify the learned parameters of a generative AI model.” That definition excludes training of models used only for non-generative tasks.

Kelley explained the heavy reliance on existing syntax on the mailing list. “I’ve spent a lot of time investigating how deployed parsers would interact with a wide variety of ways of expressing preferences in robots.txt files,” he wrote, adding that the design “avoids relying on the areas where deployed parsers behave differently (and let me tell you, there are plenty!).”

The wildcard catch most site owners will miss

One clause will matter more to most site owners than any other: “However, wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token,” the draft states. A robots.txt file with no __usage_ai-training group expresses no preference at all.

In practice, a site whose robots.txt shuts out every bot today expresses no AI-training preference under the new scheme. Owners who want out have to add the new group by name.

The signal also only looks forward. Content the companies crawled before the line appears is already in their training sets, and nothing in the draft forces any company to obey it.

Rollout dates by company, and Microsoft’s retired opt-out

The four companies did not share a start date.

Company	Stated timing	Existing controls
Microsoft	Honours the new preferences from February 1, 2027, as it crawls	Stops reading NoArchive and NoCache as training or grounding signals after March 1, 2027
Google	“In 2027” in its blog post; “early 2027” in its crawler documentation	Google-Extended keeps working alongside the shared signal
OpenAI	Aims “to implement the signal in the coming months”	Existing crawler controls stay in place and respected
Anthropic	Has “already started implementing support”; no live date given	Draft described as a complement to existing crawler controls, not a replacement

Kelley also wrote that covering all of Anthropic’s services will take time. Google says it will provide updates on its progress.

Microsoft says it was the first company to offer publisher controls for AI training and grounding, through the NoArchive and NoCache tags it introduced in 2023. Those tags lose their role as AI signals after March 1, 2027, so sites relying on them need to move to the new preferences.

What Google and Microsoft each add on top of the shared rule

Google: the opt-out also reaches Gemini grounding

When a site uses the shared token to opt out of training and has not explicitly specified Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Google-Extended is the AI training control Google says it has offered since 2023. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. “Grounding” here means a model fetching web pages to support and cite an answer it generates on the spot. In plain terms, the shared opt-out at Google will also keep a site out of Gemini answers that cite web pages, while leaving its Google Search rankings unaffected.

Existing company-specific controls keep working, so a site can state a broad choice through the shared signal and still tailor instructions to individual developers.

Microsoft: separate training and grounding switches

Microsoft is adopting the shared AI training preference for sites and paths and extending it to individual pages through meta tags and HTTP response headers. It is also adding a separate page-level AI grounding preference. A site could refuse training and still allow grounding, keeping its citations and links inside AI answers.

Alongside Allow and Disallow, the grounding preference adds a DisallowExtended option. It limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, while excluding grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry. Both preferences default to Allow when nothing is published.

Microsoft warns that refusing grounding will “significantly reduce” a site’s visibility in its AI experiences, and that neither choice guarantees inclusion, traffic or payment. The preferences do not change eligibility for traditional Bing search results.

What it means for builders and site owners

For anyone running a product site, documentation, blog or shop, the practical task is small: add one named group to robots.txt rather than a list of per-company rules. Because the token is about use rather than access, search keeps working. Teams that already use a blanket User-agent: * block should not assume they are covered, because of the wildcard clause.

For developers who build AI applications on top of grounding services, Microsoft’s DisallowExtended option is the detail to note. Publishers who choose it stay available to Microsoft’s own products but drop out of grounding offered to third parties, including grounding through API access in Foundry. The pool of pages available to third-party grounding could therefore differ from what Microsoft’s first-party assistants see.

For teams that operate their own crawlers or collect web data, the draft’s design keeps parsing simple: exact token matching apart from case, standard RFC 9309 path rules, and no new grammar. The draft also notes that private agreements may override expressed preferences.

Open questions: standards status, Meta and the EU

The draft is short and early. It runs to ten pages including a long acknowledgments table, is intended for the standards track, and expires on May 3, 2027. Per the IETF datatracker, it is an individual submission that is not endorsed by the IETF and has no formal standing in the standards process. It still has to pass through the AIPREF working group before it becomes a formal standard.

The most conspicuous absence is Meta. Its head of communications, Andy Stone, told The Verge that a common signal reaching several providers at once is “an important step,” but did not say whether Meta will honour it. On the same day, Meta’s updated business terms took effect, giving it permission to send crawlers to websites tied to Facebook, Instagram and WhatsApp business accounts and to use what they collect to train its products. OpenAI encouraged other AI providers to honour the signal.

On regulation, Google says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary method for web publishers to express machine-readable preferences. Google also said the proposal reflects feedback from publishers, civil society groups, regulators and others who have called for industry-wide controls on AI training.

What to watch next

• February 1, 2027: Microsoft begins honouring the new preferences as it crawls.
• March 1, 2027: Microsoft stops reading NoArchive and NoCache as training or grounding signals.
• Early 2027: Google’s target for supporting the token, with progress updates promised.
• OpenAI and Anthropic live dates: neither has named one yet.
• May 3, 2027: the current draft expires, so a revised version or working-group adoption is the next standards milestone.
• Meta: whether it joins the four companies.