Skip to content
Policy · Oct 30, 2026

Major AI labs to honor new web standard allowing publishers to opt out of AI scraping

OpenAI, Google, Microsoft, and Anthropic will adopt a shared 'policy token' in robots.txt to let publishers block AI crawlers at scale.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • A proposed IETF standard introduces a shared 'policy token' in robots.txt that publishers can use to signal opt-out from AI scraping.
  • OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers if the token is present.
  • Meta expressed support but did not commit to honoring the token.
  • Google and Microsoft plan to support the standard next year; OpenAI expects to implement it in the coming months.

A proposed standard within the Internet Engineering Task Force (IETF) introduces a shared 'policy token' that publishers can place in their robots.txt file to signal opt-out from AI scraping across multiple AI companies. The signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic.

OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if the shared token is present. Currently, publishers must set opt-out preferences separately for each AI company.

Meta communications head Andy Stone said the company supports creating a common signal to convey such preferences to multiple providers, but did not state whether Meta will honor the token.

Microsoft said it will offer granular controls allowing publishers to opt out of AI training or AI grounding while still appearing in Bing’s search results. Google implemented similar controls that let sites control whether their content is used for AI training without affecting search visibility.

Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year. OpenAI expects to implement the standard in the coming months.

Sources
  1. 01The Verge — AI — The biggest names in AI are backing a new way to opt out of scraping
Also on Policy

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.