If you run a website, you’ve likely noticed that automated bot traffic now outnumbers human visitors. Cloudflare recently reported that more than half of all web requests are now non-human.
That shift is forcing a fundamental question: What kind of access should AI crawlers have to your content?
Cloudflare’s response is a set of granular controls rolling out now, with a hard deadline of September 15, 2026. The catch is that the default settings will change and if you’re not paying attention, you could accidentally lock out the crawlers you actually want.
The issue isn’t that Cloudflare is broken, it’s that publishers, for the first time, have a real choice. And choices require strategy.
What’s actually changing
Cloudflare is replacing its old “block all AI bots” toggle with three distinct crawler categories:
| Category | What it does | Default from September 15, 2026 |
|---|---|---|
| Search | Indexes pages for search and cited AI answers | Allowed on ad-monetized pages |
| Agent | Acts on behalf of a user in real time (e.g. an AI assistant browsing or buying) | Blocked on ad-monetized pages |
| Training | Collects content to train AI models | Blocked on ad-monetized pages |
The logic is straightforward: if a page displays ads, it’s meant for human eyes. Cloudflare’s default keeps humans as the priority and blocks bots that don’t send visitors back.
But here’s where it gets tricky. Around a third of crawler activity now comes from mixed-purpose bots that combine two or more categories – for example, a crawler that does both search and training. Cloudflare applies the most restrictive rule to these. So if you block Training, a mixed crawler can be blocked too, potentially even Googlebot on ad-monetized pages.
This is the single biggest risk: an aggressive block can accidentally hurt your normal search visibility.
The bigger picture and why this matters
The economics of the web are changing. For decades, publishers accepted search engine crawlers because indexing drove referral traffic and ad revenue. Generative AI has disrupted that by consuming content to answer questions directly, often bypassing the need for users to visit the source.
Cloudflare’s data illustrates the imbalance starkly: OpenAI’s GPTBot made roughly 1,700 content requests for each referral click it generated. Anthropic’s ClaudeBot was even worse with about 73,000 requests per referral.
That’s not a fair trade and it’s why Cloudflare is also evolving its earlier Pay Per Crawl model into Pay Per Use, a system designed to compensate publishers when their content actually contributes value to AI-generated responses.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.”
Matthew Prince, Cloudflare CEO.
The strategic choice you need to make
There isn’t one right answer. Your decision depends on your business model:
- If you rely on ad revenue or sell content directly, blocking Training and Agent bots makes sense. You’re protecting your intellectual property from being used without compensation.
- If you’re building awareness and want maximum distribution, you might allow everything. More visibility in AI search and assistants can drive discovery, even if you don’t get direct traffic.
- If you run an e-commerce or lead-gen site, you may want to allow Agent bots. These AI assistants are increasingly browsing and buying on behalf of users.
A practical way forward
Blocking everything is rarely the answer. Rather than a blanket approach, consider this:
- Whitelist known search bots. Make sure Googlebot, Applebot, Bingbot, and other legitimate search crawlers are explicitly allowed.
- Use granular controls. Cloudflare’s BotBase, a searchable database of known bots, makes it easier to see exactly who is crawling your site and why.
- Test before the September 15 deadline. After any change, check your sitemap access and server logs. If you see 403 errors where legitimate crawlers should be allowed, adjust immediately.
The bottom line
Cloudflare’s new controls aren’t a trap. They’re an invitation to think strategically about how your content is used and by whom.
The publishers who will thrive in this new environment aren’t the ones who panic and block everything, nor the ones who stay passive. They’re the ones who take the time to understand the tradeoffs, make a deliberate choice, and revisit that choice as the landscape evolves.
If you’re unsure where your site stands, now is the time to look, before the September 15 deadline makes the decision for you.