Beckmann

Dossier · Ongoing

Cloudflare's AI Crawler Controls

From the first purpose-based split in July to the Accountable status in September 2026: Cloudflare's AI crawler controls, tracked over time.

Since July 2026, Cloudflare has been building increasingly granular controls for AI bots step by step: from the first three-way split into Search, Agent, and Training categories to the Accountable status, which decouples search visibility from AI training permission.

This thread tracks how the conflict between search engine visibility and protecting content from unpaid AI training plays out for website operators.

Timeline

  1. Cloudflare now allows AI bots to be controlled by purpose

    Cloudflare introduces differentiated AI bot controls by purpose: Search, Agent, and Training can now be controlled separately.

  2. Cloudflare separates AI training from search engine access

    A new accountable status allows website operators to refuse AI training without disappearing from Google and Bing search results.