The short version: there are two ways to be invisible to an AI shopping assistant. The one everyone talks about is bad data — thin descriptions, missing attributes, prices that contradict your feed. The one almost nobody checks is simpler and more absolute: the agent asked for your page and your own infrastructure said no. Bot management, WAF rules, rate limits and CAPTCHA challenges don't know the difference between a scraper and a shopper's assistant unless you've told them. On 1 July, Cloudflare split AI traffic into three named categories — Search, Agent and Training — and gave every customer controls over each. From 15 September, new defaults start applying automatically. If you've never made a deliberate decision about which of those three you welcome, one is about to be made for you.
What actually changed?
Until recently, AI traffic was one undifferentiated bucket: allow the bots or block the bots. That was always a bad trade, because the three things AI companies use your site for are genuinely different, and most merchants feel differently about each.
- Search — crawls and indexes your content so an assistant can answer questions about it later. This is how you get cited and recommended. You almost certainly want this
- Agent — automated behaviour acting in real time on a person's behalf. This is the shopping assistant fetching your product page mid-conversation because a customer asked. This is a customer, wearing gloves
- Training — a crawler taking your content to train or fine-tune a model. This is the one where reasonable merchants land in different places, and where the money argument lives
Cloudflare's controls now let you answer those three questions separately instead of with one switch. That's the useful part. The part with a date on it is the defaults: from 15 September 2026, for new domains, new sites added by existing customers, and existing free-plan customers, Training and Agent are blocked by default on pages that display ads, while Search stays allowed. The stated reasoning is that an ad is a signal a human was meant to land there. You can opt out in your security settings before the date.
The honest read on the deadline
If you run a normal storefront that doesn't serve display ads, the 15 September default probably doesn't touch you. We'd rather say that plainly than sell you an emergency. The reason it still matters is what it reveals: the edge is now the layer where AI visibility gets decided, defaults there are starting to move on their own, and most merchants have never looked at the settings. The deadline isn't your risk. Not knowing your own configuration is.
Why would my site block a shopping agent by accident?
Because nothing about an agent looks like a human visitor, and every control you turned on to stop bad automation is doing exactly what you asked. The volume alone changes the picture: industry measurement of agentic traffic through 2025 put its year-on-year growth in the thousands of percent, concentrated overwhelmingly in retail, media and travel. Your bot rules were tuned for a world where that traffic was rare and hostile. It's now neither.
- Aggressive bot-fighting modes challenge anything with non-human behavioural fingerprints — an agent fetching six product pages in four seconds fits the profile perfectly
- Rate limits written per-IP punish agents, which arrive from a small pool of datacentre addresses on behalf of many different shoppers
- CAPTCHA and interstitial challenges are, by design, things an autonomous agent cannot complete. A challenge page is a silent no
- Geo and ASN blocks catch cloud ranges wholesale, and agent traffic originates from exactly those ranges
- Blanket robots.txt or firewall rules added during a scraping scare, never revisited, that predate any of these categories existing
There's a second-order trap worth knowing about too: multi-purpose crawlers. Googlebot, Applebot and Bingbot do more than one of those three jobs. If you block Training, the most restrictive applicable rule can win — and you can find yourself blocking the crawler that feeds your ordinary organic search results. That is a genuinely expensive way to make a point about model training.
The failure mode we see most
A merchant spends a quarter on structured data, feed hygiene and product copy. Nothing moves. Nobody looks at the edge, because the site loads fine in a browser — and it does, for humans. The agent has been getting a challenge page for months. From the outside it's indistinguishable from being outranked, so the conclusion drawn is "AI search doesn't work for us" rather than "we never answered the door."
What about Shopify — isn't this handled for me?
Partly, and it's worth being precise about which part. Shopify storefronts sit behind Cloudflare, so platform-level decisions are made for you and you don't get to tune much of it. That's mostly a good thing. But Shopify has also been tightening its own posture: since a May 2026 change, bots and agents are expected to identify themselves using Web Bot Auth — cryptographic signatures on their requests — and unsigned traffic is subject to the strictest rate limits. Merchants can generate signatures in the admin to authorise their own crawlers and tools.
Two practical consequences. First, the identified, well-behaved agents from major platforms are the ones with the smoothest path — which is another way of saying the ecosystem is moving from "block everything suspicious" to "verify identity, then allow." Second, if you run your own SEO audits, monitoring, or an internal tool that crawls your store, sign it. Otherwise your own tooling is competing for the same throttled lane as anonymous scrapers, and your audits get quietly incomplete results.
So what should I actually do this month?
This is an afternoon of work, not a project. The goal is to replace an assumption with a measurement.
- Fetch your own product pages the way an agent would — from a cloud IP, without a browser, several in quick succession — and record the status codes. A 200 in Chrome tells you nothing about this
- Open your CDN and WAF logs and filter for known AI agent user agents. Look at what share got challenged, rate-limited or blocked, not just how many arrived
- Make the three decisions explicitly and write them down: Search yes, Agent yes, Training your call. Then confirm the configuration matches the decision
- Check whether blocking Training in your setup also catches multi-purpose crawlers, and whether that costs you ordinary search
- Review robots.txt against the actual list of crawlers operating today — most of these agents did not exist when your file was written
- On Shopify: generate Web Bot Auth signatures for any first- or third-party tool that crawls your store
- Split AI and agent-referred sessions out in analytics, so the next time you change something you can tell whether it worked
The encouraging part
This is the cheapest fix in the whole AI-visibility category. Fixing thin product data is months of work across every SKU. Fixing an over-eager firewall rule is one change, and the effect is immediate and total — the difference between being read and not being read at all. It is also the only item on the list where you get a clean before-and-after number in your own logs.
Where we fit
Most agencies handling AI visibility work on the content layer, because that's the part that looks like marketing. We come from the infrastructure side as well, which is why we look at the request path first: what an agent gets when it asks, where it gets stopped, and what your edge configuration is silently deciding on your behalf. Retaining us for AI Shopping Visibility means one team owns the whole chain — the feed, the structured data, and the CDN and WAF rules that determine whether any of it is ever seen — rather than a content team and an infrastructure team each assuming the other checked. In our experience that gap is where the quarter goes.
Sources
- Cloudflare — Your site, your rules: new AI traffic options for all customers (1 July 2026; Search / Agent / Training categories and the 15 September defaults)
- TechCrunch — Cloudflare's new policy pushes AI companies to pay for publishers' content
- Help Net Security — Cloudflare changes AI crawler access rules
- Search Engine Journal — Cloudflare's AI crawler rules can block Googlebot (the multi-purpose crawler trap)
- Shopify developer changelog — Bots and agents should identify themselves via Web Bot Auth (7 May 2026)
- Shopify changelog — Authorize custom crawlers and tools with new crawler access keys
- Fortune — AI agents are eating the web, growing by nearly 8,000% (coverage of HUMAN Security's 2026 State of AI Traffic benchmark)
