Four of the biggest AI companies now say they will respect a single AI training opt-out that website owners can set once in robots.txt. OpenAI, Google, Microsoft and Anthropic announced support for a shared “policy token” on October 30, 2026, so publishers no longer have to block each AI crawler one by one. The signal comes from a draft standard proposed by an Anthropic engineer at the Internet Engineering Task Force (IETF).
How the Shared AI Training Opt-Out Works
Today, a site that does not want its pages used to train AI models has to list every company’s crawler separately in its robots.txt file. That file has told bots what they can and cannot visit for decades, but the list of AI crawlers keeps growing. The new approach adds one shared instruction that any participating AI provider agrees to read.
The proposal was written by Kevin Kelley, a technical lead for standards at Anthropic, and builds on work from the IETF’s AI Preferences (AIPREF) working group. Google’s updated crawler documentation says the signal will use the user-agent line “__usage_ai-training” inside robots.txt. If a site uses it to opt out, each participating company treats that as a “no” for training on that content.
Company-specific controls are not going away. You can still use the shared signal for a broad choice and then fine-tune rules for a single provider, such as Google-Extended for Google.
What Each Company Has Committed To
The four companies are not moving on the same timeline, and Microsoft is going a step further than the others. Here is how their plans compare, based on their own announcements and early reporting.
| Company | What it says it will do | Timing |
|---|---|---|
| Anthropic | Honor the shared token for its crawlers | Already adopted |
| OpenAI | Honor the signal for foundational generative AI training; keep existing crawler rules | Coming months |
| Treat an opt-out as if the site had blocked Google-Extended, unless Google-Extended is set directly | Early 2027 | |
| Microsoft | Honor the training signal plus new page-level training and grounding controls | February 1, 2027 |
Microsoft Adds a Separate Grounding Control
Microsoft says it will also let publishers decide whether their pages can be used for “grounding,” which is when an AI pulls in live web content to back up an answer. A new page-level setting called DisallowExtended limits grounding to Microsoft’s own products, like Copilot and Microsoft 365, while blocking grounding services it sells to other companies. A plain Disallow blocks grounding entirely.
These new settings will replace Microsoft’s older NoArchive and NoCache tags for training and grounding. Microsoft stresses that none of these choices affect whether a site appears in Bing search results, though it warns that blocking grounding can reduce a brand’s visibility in AI answers.
Google Keeps Its Existing Controls
Google says sites that explicitly allow Google-Extended remain eligible to show up and earn clicks through grounding in Gemini Apps. The company also points out that the plan lines up with the EU’s Code of Practice for general-purpose AI models, which recognizes robots.txt as a main way for publishers to state their preferences.
Why This AI Training Opt-Out Matters for Publishers
Publishers have spent the past few years fighting a flood of AI crawlers, and the tools for pushing back have been scattered. Some sites turned to licensing schemes that charge bots for access, while network providers began blocking AI crawlers by default. We have also covered fights over quieter policy changes, like Meta’s business terms that raised fears about crawling.
A single shared signal does not solve every problem, but it cuts down a lot of busywork. It also gives regulators and publishers a common, public rule to point to when they ask whether AI companies are keeping their promises.
There are limits worth knowing about:
- The standard is still a draft, and the IETF working group has not finished it.
- It is voluntary, so only companies that sign on will follow it.
- It covers future training, not models that were already trained on your content.
- Google and Microsoft will not support it until 2027.
What This Means for You
If you run a website, blog or online store, this change could make your AI choices much simpler. Instead of tracking every new bot name, you will be able to set one rule and have the biggest names in AI, including the maker of Claude AI, follow it.
Here are a few practical steps to take now:
- Review your current robots.txt file and note which AI crawlers you already block.
- Decide whether you care more about training, grounding or both, since they are now separate choices at Microsoft.
- Keep your company-specific rules in place until each provider confirms support for the shared token.
- Watch for traffic changes if you block grounding, because AI answers that cite your pages, such as those in Google AI Mode, can send visitors your way.
If you mainly use AI tools rather than publish content, this is still worth noticing. How companies like the one behind ChatGPT gather web data shapes what their models know and how fairly creators are treated.
For more updates like this, browse our latest AI news coverage.
Sources
Primary source: Google’s AI training control announcement. Additional official details from OpenAI and Microsoft Bing. Additional reporting from The Verge and Search Engine Roundtable.
Conclusion
A shared AI training opt-out backed by OpenAI, Google, Microsoft and Anthropic is one of the clearest wins publishers have seen in the long fight over AI scraping. Anthropic has already switched it on, OpenAI plans to follow within months, and Google and Microsoft are aiming for 2027. The standard is still a draft and remains voluntary, so it will not end every dispute. Still, one rule that the largest AI companies agree to read is a real step toward giving website owners a simple, meaningful say over how their work is used.
