Major AI labs to honor new web standard allowing publishers to opt out of AI scraping
OpenAI, Google, Microsoft, and Anthropic will adopt a shared 'policy token' in robots.txt to let publishers block AI crawlers at scale.
1 source · cross-referenced
- A proposed IETF standard introduces a shared 'policy token' in robots.txt that publishers can use to signal opt-out from AI scraping.
- OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers if the token is present.
- Meta expressed support but did not commit to honoring the token.
- Google and Microsoft plan to support the standard next year; OpenAI expects to implement it in the coming months.
A proposed standard within the Internet Engineering Task Force (IETF) introduces a shared 'policy token' that publishers can place in their robots.txt file to signal opt-out from AI scraping across multiple AI companies. The signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic.
OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if the shared token is present. Currently, publishers must set opt-out preferences separately for each AI company.
Meta communications head Andy Stone said the company supports creating a common signal to convey such preferences to multiple providers, but did not state whether Meta will honor the token.
Microsoft said it will offer granular controls allowing publishers to opt out of AI training or AI grounding while still appearing in Bing’s search results. Google implemented similar controls that let sites control whether their content is used for AI training without affecting search visibility.
Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year. OpenAI expects to implement the standard in the coming months.
- Nov 11, 2026 · Wired
EU proposes Digital Fairness Act to ban dark patterns and restrict addictive design features
Trust75 - Nov 11, 2026 · AI Snake Oil — Narayanan & Kapoor
New nonprofit STABL launches to study societal bottlenecks of advanced AI
Trust79 - Nov 10, 2026 · TechCrunch — AI
New York enforces AI safety law requiring registration and incident reporting for frontier labs
Trust75