Your website now has a door policy about AI. On most sites, nobody chose it.

On the 7th of May this year, Shopify changed the rules for how machines are allowed to talk to a store. On the 15th of September, Cloudflare — which sits in front of a very large share of the small-business web — starts applying new default rules about which AI traffic gets through. Neither change arrived as an email to the person who owns the business. Both are, stripped of the jargon, decisions about who is allowed to look at your website.

If the instinct is that this is somebody else's problem — a plumbing detail for whoever built the site — I would gently push back. The setting in question decides whether the software your next customer is using can see your prices, your service area, and your booking page. That is not plumbing. That is the front door, and increasingly there is a bouncer on it that you did not hire and have never met.

What actually changed

Cloudflare now splits automated traffic into three categories rather than one: Search, Agent, and Training. From the 15th of September, for domains newly onboarding, Training and Agent are blocked by default on pages that carry ads, while Search stays allowed. Existing sites keep whatever they already have — which, for most small businesses, means a setting nobody has ever opened.

Shopify went a different way at the same problem. Since the 7th of May, bots and agents hitting a Shopify-hosted store or its storefront API are rate-limited according to whether they cryptographically sign their requests. Sign, and you get sensible limits. Turn up unidentified, and you get the strictest tier available.

Neither is a ban, and neither is aimed at small operators specifically. But read together they show the direction clearly enough: the web's default answer to an unidentified machine is moving from yes to it depends, and the platforms are deciding the it-depends on your behalf until you tell them otherwise.

The distinction nearly every article about blocking AI bots skips

Most advice on this subject reduces to a single switch labelled block AI bots, which is about as useful as a switch labelled block vehicles. Three genuinely different things are wearing the same costume.

• A training crawler takes your content to build or fine-tune a model. Nothing comes back to you, now or later. Whether that trade is acceptable is a values question, and refusing it is entirely defensible.
• A search crawler indexes you so that something can name you later. It is building the list your business needs to appear on. Blocking it is not protecting your content — it is removing your name from the directory.
• An agent fetch is not a crawler at all. It is a person, right now, who has asked software to go and look at something on their behalf. The robot is doing the walking; the intent behind it is human, present, and usually about to make a decision.

That third category is the one that matters most to a service business, and it is the one a blanket block catches by accident. An agent request has more in common with somebody standing in your doorway asking whether you cover their suburb than it does with a scraper. Turning it away is not a security posture. It is a lost enquiry with a technical explanation attached.

The honest cost on the other side

It would be dishonest to argue only one way here, because letting everything through has a real price and anyone who tells you otherwise is selling something. Automated traffic distorts your analytics, so the numbers you use to judge whether the marketing is working quietly stop meaning what you think they mean. It costs bandwidth and, on some plans, money. And there is the unglamorous one: your quote form is a public button that causes work to happen, and cheap automation makes junk submissions cheaper to send at volume. Every rubbish enquiry costs a real person a real minute deciding it is rubbish.

So the answer is not to fling the doors open. The answer is that allow-or-block is the wrong shape of control for this problem. A door that is either wide open or bolted shut cannot express the thing you actually want, which is: come in, but not two thousand times a minute, and say who you are on the way past.

Identity is arriving, and it is the genuinely interesting part

Here is the bit I find quietly remarkable. For about twenty years, the way a website told a bot from a person was by reading the user-agent string — a line of text in which the visitor describes itself. Anyone can type anything into it. Every bot rule you have ever had, every blocklist, every analytics filter, has been built on the honour system, and the only reason it half worked is that most bots could not be bothered lying.

Web Bot Auth changes the question. Requests are cryptographically signed, so a site can verify that an agent is genuinely the thing it claims to be rather than taking its word for it. It is an IETF draft led by Cloudflare; the major agent operators have implemented it, and platforms including Shopify, Vercel, Akamai and AWS have built support. The honest caveats: it is still a draft, adoption is patchy, and unsigned traffic will make up a large share of what hits your site for years yet.

For twenty years, the question of whether a visitor was a bot was answered by the visitor. Signed requests are the first time it has a real answer — and the first time turning traffic away is a choice rather than a guess.

That is why the platforms are moving toward limits based on identity rather than blanket blocks. It is a better tool. It also means the sites that end up worst off are not the ones that made a considered choice either way — they are the ones running a default from three years ago that nobody has read since.

What I would actually do about it

• Find out what sits in front of your site — Cloudflare, your host, Shopify, a WordPress security plugin — and open its AI or bot settings page. Read what it currently says. Most owners have never seen that screen. What you choose matters less than the fact that you chose it.
• Split the decision three ways instead of one. Refusing training crawlers is a defensible position. Refusing search and agent traffic is turning away the two mechanisms by which strangers currently find you.
• Make the pages an agent needs boringly legible: what you do, where you go, how the pricing works, how to book. In ordinary text, in the page itself. Anything that only appears after three clicks through a script-driven widget effectively does not exist to software reading on a customer's behalf.
• Treat your quote form as an interface for machines, because it now is one. Validate on the server, rate-limit by source, and never let a single form submission trigger something expensive with no human in between. That is good hygiene regardless of what you think about AI.
• Do not buy a bot-management product for a problem you have not measured. Look at your actual traffic logs first. For a five-page local services site the answer is usually a setting, not a subscription.
• Put a note in the calendar to look again in six months. This corner of the web is changing faster than the advice written about it.

The part that is not hype

I am not going to claim your customers are all robots now. They are not. The overwhelming majority of enquiries a service business gets are still a person, on a phone, who found you through somebody they know. That will be true for a long time, and any vendor telling you the website is about to become a machine-to-machine interface is describing a future they would like to sell you a product for.

The thing worth taking seriously is not the volume. It is the defaults. Defaults are how the web actually gets configured — not by decisions, but by nobody making one. A setting nobody chose becomes policy, and policy nobody remembers setting is the most expensive kind, because on the day it costs you something there is no obvious place to look.

So: go and open the setting. That is the whole ask, and it is ten minutes. If it turns out you are already making a sensible choice, brilliant — close the tab and get on with your day. If it turns out you are quietly refusing entry to the software your customers are increasingly sending ahead of themselves, you would rather find that out on a Tuesday of your own choosing than at the end of a quiet quarter you cannot explain.