Cloudflare Disallow AI Training: what to check

If your business website sits behind Cloudflare, the settings that decide which AI companies can use your pages changed on 15 September 2026. Cloudflare announced a new setting, Disallow AI Training, that “lets any website refuse AI training while staying fully in search results”. Cloudflare says the new controls are available “to all customers, on all plans”.

For most owners, nothing needs doing. Cloudflare says: “Nothing, in almost every case. Your current settings carry over on their own.” But one change can hurt a small business that goes looking for an “AI off” switch: choosing Block now also blocks Google’s and Microsoft’s search crawlers.

What changed

Some of the biggest crawlers do two jobs. Googlebot, Bingbot and Applebot collect pages both for search and for AI training. Cloudflare calls these “mixed-use crawlers”. Blocking them to refuse AI training also means blocking search, and until now Cloudflare offered no way to refuse training by those crawlers while keeping search.

Cloudflare is retiring its single “Block AI Bots” switch in favour of three controls, first introduced in July, set for each domain in the Cloudflare dashboard under Security Settings:

Search: crawling to build a search index.
Training: crawling to train or fine-tune an AI model.
Agent: AI tools visiting a page on behalf of a person, such as a chatbot fetching a page.

Each can be set to Allow, Block on pages with ads (crawlers are blocked only on pages detected to be serving an ad) or Block. Training also gets the new Disallow AI Training option. Under it, Cloudflare publishes a no-training instruction in your robots.txt file (the file that tells crawlers what they may do). It keeps Googlebot, Bingbot and Applebot crawling for search, and blocks every other training crawler, “including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI”.

The catch is in the other options. Cloudflare says “Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training.” On our reading, a business that sets Training to Block to keep its content away from AI models would now also be telling Cloudflare to keep Google out.

Cloudflare’s “Managed Robots.txt” feature is also being replaced by Bot Preference Sync, which writes your preferences into robots.txt. If your site already has a robots.txt file, Cloudflare says its lines are added at the top and “any existing Disallow directives are maintained”.

What happens to your current settings

Cloudflare says it will “preserve the practical effect” of existing choices:

Your old “Block AI” setting	Search	Training	Agent
Disabled	Allow	Allow	Allow
Block	Allow	Disallow AI Training	Block on pages with ads
Block on pages with ads	Allow	Disallow AI Training	Block on pages with ads

If you had already used the separate controls, Search and Agent stay as they were, and a Training setting of Block or Block on pages with ads becomes Disallow AI Training.

New domains added from 15 September are offered a preset. A site that doesn’t carry ads gets Allow for all three. A site that earns money from ads gets Disallow AI Training for Training and Block on pages with ads for Agent. Search stays allowed in both, and every setting can be changed later.

What it doesn’t cover

Bing, for now. Microsoft is building support for a robots.txt “no training” preference, “targeted for early 2027”. Until then, Cloudflare says, “selecting Disallow AI Training will not automatically convey a no-training preference to Bing”. Bing’s existing method is the NOARCHIVE tag: Microsoft said in 2023 that it won’t use content labelled that way to train its generative AI models. The content still appears in search results, but not in its AI chat answers. Cloudflare also points to Bing’s Block URLs and Content Removal tools.
Google’s AI Overviews. Google’s training opt-out is the “Google-Extended” robots.txt token, which Google says “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. It covers Gemini training and grounding, not AI Overviews. Google says a separate Search Console control lets owners decide whether their site appears in AI Overviews and AI Mode, rolled out to all websites worldwide by 31 August 2026. Sites that opt out “will not receive traffic or impressions from our generative AI features”.
Crawlers that ignore robots.txt. Cloudflare’s own documentation notes that “robots.txt compliance is voluntary”. Disallow AI Training blocks training crawlers other than Googlebot, Bingbot and Applebot; for Google and Apple, it relies on each company honouring the robots.txt instruction.

Apple says sites that disallow its training crawler, Applebot-Extended, “can still be included in search results”.

Should a small business refuse AI training?

That’s a business call, not a technical one. Cloudflare’s own data suggests most sites want to be found: “fewer than 1% block search crawlers, while 17% restrict AI training”.

On our reading, for a typical service business whose website exists to win enquiries, staying visible in search matters far more than the training setting. Businesses that publish original material they sell or license, such as detailed guides, photos or course content, have more reason to choose Disallow AI Training. Either way, avoid Block unless you really want to stop Google and Bing crawling your site for search. If you’re moving hosts or changing Cloudflare accounts, our guide to moving a website without losing rankings covers the search checks.

Checklist

Find out if you use Cloudflare. Ask your web developer or IT provider whether your website runs through a Cloudflare account, and who holds the login.
Open Security Settings in Cloudflare. Check what Search, Training and Agent are set to for your domain.
Confirm nothing is set to Block unless you mean it. Block on Training now also stops Googlebot, Bingbot and Applebot, and Block on pages with ads stops them on ad pages.
Choose Disallow AI Training if you want to refuse training. It keeps the big search crawlers while telling AI companies not to train on your pages.
Check your robots.txt after any change. Visit yourdomain.com.au/robots.txt and confirm your own Disallow lines are still there.
Decide on AI Overviews separately in Search Console. Opting out there means no traffic or impressions from Google’s AI search features.
Ask your developer about NOARCHIVE if you want Bing to skip training now. Cloudflare’s setting won’t reach Bing through robots.txt until Microsoft’s planned early 2027 support, and NOARCHIVE also keeps pages out of Bing’s AI chat answers.

Download checklist PDF

Sources 9 sources

What changed

What happens to your current settings

What it doesn’t cover

Should a small business refuse AI training?

Checklist

The week in digital, every Friday

Related stories

.au domain names: renewals, transfers and the direct .au, explained

auDA's plan to tighten .com.au rules: what it means for your domain, and why a petition is fighting it

Website accessibility: what the Disability Discrimination Act means for your site