Google, Microsoft, OpenAI, and Anthropic Support Shared robots.txt Token to Opt Out of AI Training

On October 30, Google, Microsoft, and OpenAI announced support for a shared policy token in robots.txt allowing websites to express preferences against AI training in a single configuration, with Anthropic also rolling it out according to AFP. While designed to eliminate the need for separate rules for individual AI crawlers, the setting only addresses model training rather than crawling activities. The mechanism originates from an individual draft submitted to the IETF and lacks official standard endorsement, with OpenAI describing it as a voluntary measure. Private agreements may supersede the preference, and existing crawl policies remain valid. Microsoft plans adoption in February 2027, Google targets early 2027, and OpenAI plans rollout over coming months. Google noted opting out will also disable grounding in Gemini Apps, whereas Microsoft separates training controls from search grounding.

Key takeaways

• Major AI vendors support a shared robots.txt token (User-Agent: __usage_ai-training) to opt out of AI training, requiring explicit specification rather than wildcard rules.
• Originating from an IETF draft without official standard endorsement, the token addresses future model training parameters rather than blocking web crawling.
• Adoption timelines vary: Microsoft plans rollout by February 2027 while separating grounding, whereas Google targets early 2027 and links the opt-out to Gemini grounding exclusion.