Cloudflare sets September 15 deadline for AI crawlers to separate search from training use

Cloudflare announced a September 15, 2026 deadline for AI companies to separate the web crawlers used for traditional search from those used for AI training and agents. Starting on that date, Cloudflare’s default settings will block 'mixed-use' crawlers from any pages that host ads unless the site owner adjusts the settings.

The policy applies to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Cloudflare co-founder and CEO Matthew Prince framed the move as necessary to address the milestone where non-human bot traffic surpassed human traffic online, a shift that Cloudflare noted was not expected until 2027.

Cloudflare’s announcement highlights tensions between publishers and AI companies over access to content. Publishers often want their content discoverable via search and AI services but seek protections against giving away intellectual property for free. Cloudflare cited concerns that some AI companies access significantly more content than others, referencing what it described as the 'world’s largest search engine' having access to '2x more information' than other AI companies.

The company also introduced or expanded tools to give publishers more control over AI-driven access to their content. Cloudflare’s 'Pay Per Crawl' marketplace, which lets websites charge AI bots for scraping, is evolving into 'Pay Per Use.' This new model aims to let publishers charge AI companies when their content creates value, not just when it is fetched. Cloudflare is initially working with partners Ceramic.ai and You.com to implement this model, with publishers paid when their content appears in Ceramic.ai’s AI search results or when You.com accesses premium content.

Cloudflare’s data suggests that over 50% of crawl traffic from AI crawlers involves re-fetching unchanged pages, which the company argues wastes publishers’ bandwidth and compute resources. The policy and tools are designed to encourage AI companies to adopt clearer, more sustainable crawling practices while providing publishers with commercial opportunities.