If you run an ad-supported site, you’ve likely felt a specific kind of bind for a while now: block AI crawlers and risk limiting search visibility, or allow them and risk having your content absorbed into a model that competes with you for the same reader’s attention, with nothing coming back to you either way.
That bind hasn’t been solved, but it’s finally being addressed at the infrastructure level. It’s worth understanding what’s shifting and how to respond, regardless of which CDN or security layer you sit behind.
Cloudflare, which sits in front of a large share of the web, announced an update on July 1, 2026 that splits AI bot traffic into three purposes instead of one blanket “AI or not” toggle: crawlers that build a searchable index of your pages, automated agents completing a task for a person in real time, and crawlers pulling content to train a model.
Starting September 15, 2026, new publisher sites onboarding to Cloudflare will, by default, block agent and training traffic on ad-supported content while continuing to allow search-oriented crawlers. Crawlers with multiple purposes will now be judged on everything they do rather than getting a pass because one of their jobs is search indexing.
Cloudflare is also evolving its pay-per-crawl marketplace toward a newer “Pay Per Use” model, aiming to compensate publishers based on how their content is actually used by AI systems rather than simply when a bot fetches the page.
The biggest change isn’t that publishers suddenly have more ways to block AI. It’s that tools are beginning to give publishers more granular control over how and why AI systems access their content. Instead of treating every AI crawler the same, publishers can increasingly make decisions based on whether that access is likely to drive discovery, generate revenue, or simply extract value. That distinction becomes especially important when a single crawler serves multiple purposes, such as search indexing alongside AI training or agent activity. Publishers therefore need to understand what a specific crawler actually does—and how their infrastructure classifies it—before blocking it outright. While this update comes from Cloudflare, it reflects a broader shift toward more purpose-based AI traffic management across the industry.
Not all bot traffic is equal, and treating it that way has cost publishers
The industry is finally catching up to something publishers have been saying for a while: a crawler that eventually sends you a reader is fundamentally different from one that reads your page once and no human ever sees the result. Until recently, most tools forced a binary choice, either block everything and lose discoverability, or allow everything and hand over training data for free.
Splitting traffic by purpose, whichever vendor does it, is the first real acknowledgment that these are separate decisions, not one.
Ad-supported pages carry the most exposure
This distinction matters most on your ad-supported pages specifically. A visit from a bot that’s only there to train a model or complete an unrelated task is pure infrastructure and CDN cost, with no chance of an ad impression, a session, or a return visit attached to it.
Search-oriented traffic, even when imperfect, at least has a plausible path back to a human reader. That gap in outcome is why ad pages deserve a different default than the rest of a site.
Cloudflare’s three categories offer a useful starting lens for thinking about AI traffic generally, since the underlying logic behind them, weighing what a bot costs you against what it could realistically give back, applies to any AI crawler regardless of who’s classifying it.
Search and answer-indexing crawlers carry a relatively low direct cost today, though that may shift as more answer engines summarize content without sending a click through. Their upside is still real: referral traffic, discoverability, and occasional attribution. For most ad-supported pages, it’s often reasonable to keep these allowed, while tracking click-through and referral rates by source rather than assuming every search-labeled crawler still behaves the way it used to.
Agent or task-completion bots can consume ad-monetized pageviews with no human attending them, and may complete a task without a single ad ever being seen. Unless there’s a direct payment or transaction tied to that visit, the upside is generally minimal, which is why blocking these by default on monetized pages, then evaluating exceptions case by case, tends to make more sense than leaving them open.
Training crawlers absorb content into a model with no ongoing return, and carry a longer-term risk that the model itself starts replacing the direct visits a site used to get. There’s little to no upside here unless it comes attached to an actual licensing or usage-based arrangement, which makes blocking by default a reasonable starting position, with exceptions reserved for crawlers you have a paid deal with.
Even on infrastructure that doesn’t offer this exact three-way split, the same questions are worth asking of any bot hitting your site: what is it actually doing with your content, and is there any plausible path back to you as a result.
This shift touches both sides of the business: what you’re spending on unwanted traffic today, and how you plan for AI’s growing role in how readers find you tomorrow.
The revenue side
Non-human visits generally don’t translate into meaningful ad revenue and can consume infrastructure resources while inflating traffic metrics. Publishers running header bidding or programmatic setups often don’t realize how much of their raw traffic volume, and therefore their reported page views, is inflated by crawlers with no chance of ever seeing an ad.
Cutting that traffic doesn’t just reduce cost, it also cleans up the data feeding into your ad stack, which matters for yield management, viewability reporting, and any deal terms tied to traffic quality. A site that quietly reduces its bot load on monetized pages may improve the accuracy of reporting and, in some cases, contribute to stronger monetization metrics over time.
The traffic strategy side
The bigger question is longer term. As more of the open web’s discovery layer shifts from traditional search results to AI-generated answers, publishers need a point of view on which of that traffic they want to encourage and which they want to charge for.
Allowing search-purpose crawlers keeps you visible in whatever discovery layer wins out, whether that’s classic search or an AI answer engine that still sends attribution. Training crawlers and most agent traffic offer no such path back, so treating them identically to search traffic, as many default settings still do, is very likely giving away value for nothing.
Usage-based compensation models are still early and only cover a handful of partners today, so this isn’t a revenue line to plan around yet. But publishers who start tracking and segmenting this traffic now, rather than waiting for a mature market to form, will have both better data and a head start when more AI platforms start paying for access.
The net effect is that AI bot management is no longer purely a security or infrastructure decision. It’s becoming an ad revenue and audience strategy decision, and treating it that way now puts you ahead of publishers still running a blanket block-or-allow rule they set years ago.
What is an AI crawler or bot?
An AI crawler is an automated program that visits websites to collect content, either to help answer questions later, to complete a task on behalf of a user, or to train a machine learning model. The distinction between these purposes is what determines whether the traffic sends you value back or simply takes it.
How can I tell if AI bots are visiting my site?
Most CDN and security dashboards log bot traffic separately from human visits and often identify bots by name or declared purpose. If your current platform doesn’t break this out, server logs, CDN dashboards, or bot-management reports.
Does blocking AI bots hurt my search rankings?
Blocking a bot that’s purely used for training or agent tasks shouldn’t affect search visibility, since that’s a separate function. The risk is blocking a bot that handles both search indexing and another purpose at once, which is why understanding what a specific bot actually does matters before blocking it outright.
Is robots.txt still useful for managing AI access?
Yes, though it’s worth remembering it expresses a preference rather than an enforced rule. Well-behaved bots respect it, but it won’t stop ones that ignore the standard, which is why many publishers now pair it with server or CDN-level bot management for actual enforcement.
Can publishers realistically get paid when AI platforms use their content?
It’s becoming possible, though still early and limited to a small number of marketplaces and named partners. More usage-based compensation models are emerging, but terms, payout structures, and available partners vary widely and are likely to keep changing over the next year.
Product Marketing @ MonetizeMore
10X your ad revenue with our award-winning solutions.
