Shared AI training opt-out in robots.txt: what to know

If you’d rather your website didn’t help train AI models, you’ll soon be able to say so once instead of company by company. Google and Microsoft both announced on 30 October that they plan to read one shared signal in robots.txt. That’s the file that tells crawlers what they may do on your site.

Nothing changes yet: Bing starts reading it on 1 February 2027, and Google aims to support it during 2027.

One line instead of a growing list

Until now, refusing AI training meant separate instructions for each company. The new signal comes from a draft proposal at the IETF, still a work in progress. It uses a shared “policy token” in the User-Agent line rather than a crawler’s name.

Microsoft’s announcement gives the line for a whole site:

User-Agent: __usage_ai-training
Disallow: /

You can also refuse training for one folder by putting its path after Disallow. Microsoft says that if you add nothing, Bing treats training as allowed.

What Google will do with it

Google says it will treat an opt-out through the shared token as a Google-Extended block. That applies when the site doesn’t mention Google-Extended separately. That’s Google’s own AI training control, offered since 2023.

Company-specific controls keep working alongside it, and they win for that company. Google says sites that specifically allow Google-Extended stay eligible to appear in Gemini app answers. But Google-Extended covers Gemini training as well as those answers, so allowing it lets Google train on the site too. It doesn’t affect whether your site appears in Google Search, as our AI Overviews explainer sets out.

Bing splits training from AI answers

Microsoft goes further. It is adding a separate page-level setting for “grounding”. That means using your page to help write an AI answer, as in Copilot or Bing’s AI Overview.

It has three choices:

Allow: the default. Your content can help inform AI answers.
DisallowExtended: only Microsoft’s own products, such as Copilot and Microsoft 365, can use it. Grounding services Microsoft sells to other companies can’t.
Disallow: your content isn’t used for AI answers.

Grounding is set on each page, with a meta tag or a server header, not in robots.txt. Microsoft says limiting or blocking it will “significantly reduce” your visibility in AI answers. Neither training nor grounding settings change your eligibility for ordinary Bing results. They also don’t guarantee traffic or payment either way.

That split means a site can refuse training and stay eligible to be cited in Copilot answers.

Using NoArchive or NoCache? There’s a deadline

Microsoft’s old controls were the NoArchive and NoCache tags. The new preferences replace them. After 1 March 2027, Bing will no longer read those tags as a training or grounding preference.

If your site uses either tag, Microsoft asks you to publish the new training and grounding preferences before 1 March 2027. To keep pages out of AI answers in the meantime, it also asks you to record a temporary grounding preference in Bing Webmaster Tools. It opens on 15 December 2026 for sites using the old tags. It covers pages Bing hasn’t recrawled yet, and lasts until the page is recrawled or 30 September 2027, whichever comes first.

Should you refuse AI training?

That’s a business call. Our story on Cloudflare’s AI crawler settings covers the trade-offs, and how a “Block” option can shut out search crawlers too. The new line only covers training, not search, so it won’t do that.

Like all robots.txt rules, it’s a request. It works only for companies that choose to honour it.

Checklist

Ask your developer whether your site uses NoArchive or NoCache. If it does and you want to keep refusing Bing’s AI use, set the new training and grounding preferences before 1 March 2027.
If you use NoArchive or NoCache, record a temporary grounding preference in Bing Webmaster Tools. It opens on 15 December 2026 and keeps pages out of AI answers until Bing recrawls them. Publish the on-page grounding preference as well, before 1 March 2027.
Decide whether you want to refuse AI training. Microsoft says Bing treats training as allowed if you publish no preference.
Add the shared training line to robots.txt if you do. User-Agent: __usage_ai-training followed by Disallow: / covers the whole site for providers that honour it.
Check Google-Extended in your robots.txt. If it explicitly allows Google-Extended, Google can still train on your site even with the shared line.
Set Bing’s grounding tag only if you mean it. Disallow keeps a page out of AI answers, including Copilot. DisallowExtended blocks only grounding services Microsoft sells to other companies.