Cloudflare Is Blocking AI Crawlers by Default: What It Means for Your Site

If your domain joined Cloudflare after July 1, 2025, AI crawlers are very likely blocked and nobody sent you a notification. Cloudflare switched new domains to deny-by-default that day, added a robots.txt vocabulary for content use in September 2025, and from September 15, 2026 it starts blocking AI training and agent traffic by default on pages that carry ads. The right response is not to switch everything back on. It is to pick the crawlers you want by name, and then verify that the requests wearing those names are genuine.

What Cloudflare actually changed, and when

July 1, 2025. Cloudflare announced Content Independence Day and became the first large infrastructure provider to block AI crawlers by default. Its press release that day described the shift plainly: new domains on the network deny AI crawler access unless the owner explicitly enables it, moving the model from opt-out to opt-in. Cloudflare puts its own share of web traffic at roughly 20 percent, so a single default change moved a large slice of the web at once. More than a million existing customers had already clicked the one-button block since September 2024. Pay Per Crawl shipped alongside it in beta, letting owners answer a crawler with HTTP 402 Payment Required instead of content.

The argument Cloudflare made was a ratio. Measuring June 2025 traffic, it put the crawl-to-referral ratio at 1,700:1 for OpenAI and 73,000:1 for Anthropic, against 14:1 for Google search. Enormous fetching, very little traffic sent back.

August 28, 2025. The AI Audit beta became AI Crawl Control and went generally available: a per-crawler view of who fetched what, allow or block switches for each operator, and custom 402 responses on paid plans.

September 24, 2025. Cloudflare published the Content Signals Policy, released under CC0 with a generator at ContentSignals.org. It adds a line to robots.txt describing how content may be used after it has been fetched, which robots.txt never covered. Three signals, each set to yes, no, or left unset:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Cloudflare wrote that policy into managed robots.txt for more than 3.8 million domains. Where training was already blocked, the default it published was search=yes, ai-train=no, with ai-input deliberately left unset rather than denied.

July 1, 2026. Cloudflare split bot behaviour into three classifications and made them available to every customer including the free tier: Search (collecting and indexing content to answer questions later), Agent (acting in real time on a person's behalf), and Training (taking content to train or fine-tune a model). It also set a date. On September 15, 2026, Search stays allowed by default while Training and Agent are blocked by default on pages that display ads. Those defaults apply to new customers, to new sites added by existing customers, and to free-plan accounts that have not changed the setting before the deadline. A crawler that mixes purposes without separating them is handled by the most restrictive rule that applies to any of its behaviours. Content Signals gained a fourth parameter, use=, with the values immediate, reference and full. And Pay Per Crawl became Pay Per Use, paying when content actually surfaces in an answer rather than when a bot fetches a page, launching with Ceramic.ai and You.com.

The cost that never shows up in your analytics

A blocked crawler produces no error page you will ever look at. No bounce, no 500, no support ticket. The assistant simply answers from someone else's page, and you find out months later when a competitor is the one being quoted.

That matters more each quarter. Cloudflare's own August 6, 2026 post on answer engine optimisation noted that fewer than half of all HTML page requests now come from a human. If retrieval bots cannot reach your documentation, your pricing page or your comparison content, you are absent from the source pool that answer engines draw on. Blocking a training crawler and blocking a retrieval crawler feel like the same switch in a dashboard. They are not the same decision, and only one of them protects anything you care about.

The distinction worth internalising: training crawlers take your content once and give nothing back per request. Retrieval and agent traffic is a reader arriving with a question right now. GPTBot trains. OAI-SearchBot indexes for answers. ChatGPT-User fetches a page because a person asked about it thirty seconds ago. Same company, three tokens, three very different trades.

robots.txt is a request. The edge is a decision.

This is the line that most site owners blur. Disallow: / in robots.txt is a notice posted on the door. Well-behaved operators read it and turn around. Everybody else reads it as a list of interesting paths, or does not read it at all. Content Signals is the same category of thing: a clear, machine-readable statement of intent, and its value is legal and social rather than technical.

Cloudflare's controls sit one layer down. The request is refused at the edge before your origin sees it, regardless of what the client believed about robots.txt. That is genuinely useful, and it is also why the default matters so much. A preference you never expressed is now being enforced on your behalf.

The practical consequence: your robots.txt and your edge rules must say the same thing. A site that publishes Allow: / for PerplexityBot while Cloudflare blocks it at the edge is sending a mixed message to an operator who is measuring compliance.

Allow the crawlers you want, by exact token

Do not match on the substring "bot". Do not match on "GPT". Allow specific tokens, because the tokens carry the purpose:

Cloudflare's own per-bot classification is what its enforcement uses, so check AI Crawl Control before assuming a mapping. Our AI crawler directory carries the exact User-Agent tokens, operators, published IP range files and robots.txt behaviour for 150 agents, which is what you need to write rules that do not accidentally catch the wrong bot.

Google-Extended is the clearest example of why tokens matter. It never sends a request. It exists purely so you can opt out of training use while Googlebot keeps indexing you normally. Blocking it at the edge accomplishes nothing, because there is nothing there to block.

Then verify what actually arrives

Allowing GPTBot by User-Agent means allowing anything that types GPTBot into a header. Scrapers have understood this for years, and an allowlist built on strings is an invitation. Every allow rule needs a verification step behind it.

Most major AI operators now publish JSON IP range files, and the older search crawlers support forward-confirmed reverse DNS. The method differs per operator, which is exactly the annoying part: Googlebot and Bingbot resolve, GPTBot and PerplexityBot publish ranges, Bytespider and CCBot publish nothing you can check. See verifying Googlebot for the reverse DNS pattern in detail.

Where a claimed crawler cannot be verified, the fallback signals are the ordinary ones: is the address a datacenter range that does not belong to the operator it claims, does the TLS fingerprint match the client it claims to be, are the headers consistent. Agentscan does this per request, forward-confirming verified crawlers, matching published ranges, and fusing IP origin with JA4 and headless tells to return human, known bot, AI agent or malicious automation. Verdicts are cached, so repeat checks come back in under 50ms.

Bottom line

Cloudflare has moved the default twice, and the second move lands on September 15, 2026. Check what your zone is set to now rather than what you assume it is. Decide separately about training, retrieval and agent traffic, because collapsing them into one switch is what quietly removes you from AI answers. Publish your intent in robots.txt with Content Signals, enforce the matching policy at the edge, allow the crawlers you want by exact token, and verify every one of them against published ranges or reverse DNS instead of a header anyone can type.