Stop AI training on your website: the settings that work

Your website can help train the next AI model unless you say otherwise. The good news: you can usually say no to training and still be found in Google, Bing and ChatGPT search.

The catch is that every company has its own switch. This page lists them, says what each one does and doesn’t do, and is kept up to date as they change.

Three different things AI companies do with your pages

It helps to separate three uses, because each company controls them separately:

Training: your pages become part of what a future AI model learns from.
AI answers: an AI tool reads your page to write an answer right now, often with a link back to you. Google’s AI Overviews and ChatGPT search work this way.
Ordinary search: the classic list of links.

Most of the settings below refuse training only, which leaves your site in search.

Robots.txt: one file, a line per company

Most of these controls live in robots.txt, the small file at yourdomain.com.au/robots.txt that tells crawlers what they may do. Each AI company publishes the name to put in it for training:

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

Every rule is a request. Companies that choose to honour it will. Cloudflare’s own documentation puts it plainly: “robots.txt compliance is voluntary”.

Google: training and AI answers are separate switches

Google-Extended is Google’s training control. It isn’t a separate crawler, just a name for your robots.txt. Blocking it refuses training of future Gemini models. It also keeps your pages out of answers in the Gemini app and Google’s Vertex AI grounding service.

Google says Google-Extended doesn’t affect your site’s inclusion in Google Search and isn’t a ranking signal.

AI Overviews and AI Mode have their own switch, in Search Console. Go to Settings, then Search generative AI, and choose include or exclude. Google says excluding usually takes a day or two, sometimes longer, and means no appearances in, or traffic from, AI Overviews, AI Mode and Discover’s AI features. It doesn’t affect the rest of Search.

That switch doesn’t stop training. Google says to use Google-Extended for that. Our AI Overviews explainer covers what leaving those features costs.

Bing: new controls from February

For now, Bing’s controls for AI use are its older NoArchive and NoCache tags. From 1 February 2027, Bing will honour the new shared training line (below). It will also honour a page-level tag for AI answers, which Microsoft calls grounding. Neither changes whether you appear in ordinary Bing results. Our story on the shared signal has the details and the tag wording.

If your site uses Bing’s old NoArchive or NoCache tags, note the date: after 1 March 2027, Bing stops reading them for training or AI answers. The new preferences replace them.

OpenAI and Anthropic: separate names for search and training

OpenAI uses GPTBot for content that may train its models, and OAI-SearchBot for ChatGPT search. It says each setting is independent. So you can block GPTBot and still show up in ChatGPT search. Pages a person asks ChatGPT to open, through ChatGPT-User, may not follow robots.txt rules.

Anthropic works the same way. ClaudeBot collects training data. Claude-SearchBot indexes for search, and Claude-User fetches pages people ask about. Anthropic warns that blocking those two can reduce your visibility in Claude’s answers.

Meta: a training crawler, and business terms on hold

Meta’s crawler page says meta-externalagent crawls for uses such as training foundation AI models. Another, meta-webindexer, crawls to improve Meta AI’s search results. Meta notes that robots.txt changes can take up to 24 hours to apply.

Meta’s updated Commercial Terms would have given it permission to crawl websites linked to its business users. We covered them in October. Meta’s terms page now says those updates are on hold and the existing terms remain in effect until further notice.

Cloudflare: pick Disallow, not Block

If Cloudflare runs your site, you can manage all this from its dashboard. Go to Security, then Settings, filter by Bot traffic and open the AI Crawler and Agent access policies. There are separate controls for search, agents and training.

Be careful with Block. Since 15 September, Cloudflare says Block also stops Googlebot, Bingbot and Applebot, search included. To refuse training and stay in search, use Disallow on the training control. Our Cloudflare story covers the trade-offs.

Cloudflare’s robots.txt setting, on all plans, can write the no-training lines for you. Its managed list includes Google-Extended, GPTBot, ClaudeBot and meta-externalagent. Cloudflare says Disallow won’t pass a no-training preference to Bing through robots.txt until Bing starts reading the shared signal.

One line for several companies is on its way

A draft standard would let you refuse training once instead of company by company:

User-Agent: __usage_ai-training
Disallow: /

It’s an early individual draft, not an agreed standard. Anthropic says it began recognising the signal on 30 October 2026, rolling out over time. Bing starts on 1 February 2027. Google’s crawler page says early 2027, and Google will treat it as a Google-Extended block when you don’t name Google-Extended separately. Google and Anthropic say a rule naming their own crawler wins.

Checklist

Open your robots.txt. Visit yourdomain.com.au/robots.txt and see which crawlers it already names.
Decide on training. If you’d rather AI companies didn’t train on your site, add Disallow rules for Google-Extended, GPTBot, ClaudeBot and meta-externalagent.
Leave the search crawlers alone. Don’t block Googlebot, Bingbot, OAI-SearchBot or Claude-SearchBot unless you’re happy to lose that search traffic.
On Cloudflare, use Disallow for training. Under Security, Settings, filter by Bot traffic, then AI Crawler and Agent access policies, avoid Block if you want to stay in Google and Bing.
Only exclude AI Overviews on purpose. The Search generative AI setting in Search Console removes you from AI Overviews and AI Mode, and doesn’t stop training.
Check for NoArchive or NoCache tags. If your site uses them to keep pages out of Bing’s AI, set Bing’s new preferences before 1 March 2027.
Recheck in February 2027. Bing starts reading the shared __usage_ai-training line on 1 February 2027, and Google plans to in early 2027.