Four AI Giants to Support Unified Training Preference Signal via robots.txt

Google, OpenAI, Microsoft, and Anthropic will support a common AI training preference signal, enabling webmasters to declare in robots.txt whether their web content may be used for generative AI model training. This mechanism aims to eliminate the need to maintain distinct crawler rules for each individual AI provider. Originating from IETF discussions, it is currently an Internet-Draft and pending formal standardization. The framework uses '__usage_ai-training' as a shared User-Agent identifier with standard Disallow or Allow directives. It distinguishes content usage intent from crawler indexing, ensuring that opting out of AI training does not delist pages from search engines. Anthropic has begun implementation, OpenAI expects rollout within months, Microsoft targets February 2027 alongside AI Grounding updates, and Google plans adoption in 2027.

Key takeaways

• Four major AI firms will support the unified '__usage_ai-training' tag in robots.txt to govern content permissions for generative AI training.
• The standard regulates content usage rather than crawling permissions, preserving regular search indexing while honoring training opt-outs.
• Anthropic has initiated implementation, OpenAI plans support within months, and Microsoft and Google plan rollouts in 2027.
• Originating from IETF working group discussions, the protocol remains an Internet-Draft and relies on implementation compliance by AI vendors.