Cloudflare provides all customers with new tools to more precisely control access from AI bots to their websites. Instead of only choosing between “block all AI crawlers” or “allow,” website operators can now differentiate based on the purpose of a bot.
Three Categories Instead of All-or-Nothing
According to the official blog post from Cloudflare, the new system distinguishes between three categories: “Search” for bots that index content to later respond to search queries; “Agent” for automated systems that act in real-time on behalf of a person, such as chat-fetch bots like ChatGPT users or browser agents like Gemini or Claude; and “Training,” meaning crawlers that collect content for training or fine-tuning AI models. Multi-purpose crawlers that fulfill several of these functions simultaneously will, according to Cloudflare, be captured in all applicable categories at once, rather than just one. The new settings are also available to customers on the free plan, according to Cloudflare.
New Default Settings Starting September 15
Starting September 15, 2026, new default rules will apply to all newly added domains to Cloudflare: On pages with advertisements, training and agent bots will be automatically blocked, while search crawlers will continue to be allowed by default. Cloudflare justifies this by stating that advertising is a signal that website operators consider human attention as a business foundation – a goal that training and agent bots could potentially undermine, while search traffic generally brings visitors back to the site. Multi-purpose crawlers like Googlebot, Applebot, or Bingbot will be treated according to the strictest applicable rule: Those who block training bots will automatically block crawlers that combine training with other functions. Website operators can disable these new default settings in their security settings at any time before the deadline, according to Cloudflare.
Content Use: How Bots May Use Content
In addition to the purpose category, Cloudflare introduces a second control layer that regulates what a bot may do with crawled content afterward. Three gradations are available: “immediate” allows interaction but no saving or reusing; “reference” – the default setting – allows indexing, excerpts, and backlinks; “full” allows summarization and complete reproduction. These values can be combined with the bot categories, for example, to allow search, SEO, and advertising review bots, but only up to the usage level “reference.” To facilitate this, Cloudflare expands the existing robots.txt standard “Content Signals” with an additional optional field called “use.”
BotBase and New Rules for “Verified” Bots
For enterprise customers, Cloudflare also introduces BotBase, a searchable database of all known bots directly in the dashboard, which reveals the classification and content usage of each individual bot. At the same time, Cloudflare changes the meaning of the “Verified” status: Until now, all verified bots were automatically allowed, but in the future, the respective category of a bot will determine whether and for what it receives access. A bot that fully reproduces content instead of just referencing it can, according to Cloudflare, no longer receive the “Verified” status.
Context: Bots Surpass Human Traffic for the First Time
Cloudflare’s initiative comes at a time when the relationship between human and automated web traffic has fundamentally shifted. Cloudflare CEO Matthew Prince stated, according to a report from Tom’s Hardware, in early June 2026, that automated systems are for the first time in internet history responsible for a larger share of HTTP requests than humans – specifically 57.5 to 42.5 percent according to Cloudflare’s own radar data. Prince had previously expected this turning point for the end of 2027; he admitted to reporters that the underlying classification is “somewhat messy,” but the overall trend is clear.
Classification
Cloudflare’s new three-tier category system responds to a problem that website operators have increasingly faced since the rise of generative AI: A blanket block of all AI bots often also costs visibility in search engines, while a blanket allowance means uncontrolled access to one’s own content for AI training. Whether the new default settings on September 15 will actually prove to be a practical middle ground will only become apparent when Cloudflare’s more than 20 percent market share among website operators begins to make greater use of them.
