Written by Jeremy Souffir Founder, JTS Tech Services
The direct answer first. On 30 October Microsoft and Google each announced that they will honour a single shared line in robots.txt, written as User-Agent: __usage_ai-training, that tells AI companies not to use your pages to train their models. Microsoft honours it from 1 February 2027; Google says 2027, and its crawler documentation says early 2027. Microsoft went further and separated two things that its noarchive and nocache tags have bundled together since 2023: whether your content trains a model, and whether it is quoted in an AI answer. Those are now two independent settings, and both default to Allow. Google did not split them. Its shared-line opt-out is read as a Google-Extended disallow, and Google-Extended covers both training and grounding in the Gemini app. So the same line, pasted into the same file, leaves you in Copilot's answers and can remove you from Gemini's. Nothing changes until February. The decision, and one deadline on 1 March 2027 for anyone using noarchive today, needs making before then.
What did Microsoft actually announce?
The Bing post is written for publishers, but every mechanic in it applies to a product page or a pricing page just as much as to a news article. The pieces that matter:
- Two preferences, not one. AI Training covers using a page to change the learned parameters of a generative model. AI Grounding covers using a page as input to an AI-generated answer in Bing, Copilot and Microsoft 365. Microsoft's own wording: indexing and ranking pages in traditional search results is not grounding.
- They are independent. Microsoft gives the example itself: you can set AI Training to Disallow and AI Grounding to Allow. Refusing to train a model no longer means disappearing from the answer.
- Training is set in robots.txt for the whole site or a path, with the shared User-Agent: __usage_ai-training group, and can be overridden on a single page. Grounding is page-level only. There is no site-wide robots.txt grounding preference, so it goes in a meta tag, meta name="Bingbot" content="ai-grounding:disallow", or an X-Robots-Tag response header.
- Grounding has a middle setting. DisallowExtended keeps your content available to Microsoft's own products but withholds it from the grounding services Microsoft sells to third parties, which the post names as Web IQ and the grounding API in Foundry.
- Both default to Allow when nothing is published. Microsoft also says plainly that choosing DisallowExtended or Disallow will significantly reduce your content's visibility in AI-generated experiences, without affecting eligibility for Bing search results, and that neither choice guarantees inclusion, traffic, or payment.
- These settings govern use, not access. An Allow does not override a robots.txt rule that blocks Bingbot from crawling. If your firewall or robots file already turns the crawler away, none of this is reached.
What happens to noarchive and nocache?
This is the part most sites will feel without having read either announcement. In September 2023 Microsoft said that content tagged noarchive would not be included in Bing Chat answers, not be linked to in the answers, and would not be used for training, while nocache allowed only the URL, title and snippet. That rule is still in force today. If your site carries a noarchive tag, whether because a theme shipped with it, an SEO app added it to stop cached copies, or someone followed a 2024 "block the AI" checklist, Copilot has been answering questions about your category without you.
Microsoft is retiring that arrangement on a schedule:
- 15 December 2026: temporary grounding transition preferences open in Bing Webmaster Tools, only for sites using noarchive or nocache on that date. Everyone else needs no transition step.
- 1 February 2027: the new preferences take effect as Microsoft crawls. Once it finds a new AI Training or AI Grounding preference anywhere on a site, it disregards any noarchive and nocache that remain.
- 1 March 2027: after this date noarchive and nocache are no longer read as a preference for training or grounding. A site that wants to keep its exclusion must have published the new page-level grounding preference and, for grounding, recorded a temporary preference in Bing Webmaster Tools. Pages with no recorded temporary preference default to Allow for grounding until Microsoft recrawls them.
- 30 September 2027: the temporary preferences expire. Each one lasts until recrawl or that date, whichever comes first.
For most stores this is a quiet upgrade
If noarchive is on your pages for a reason that had nothing to do with AI, you have been paying for it in Copilot since 2023 without knowing. From March 2027 Microsoft stops reading it that way and the default is Allow, so you re-enter the answers without touching anything. The only sites that have to act are the ones that want to stay out.
Publishing a preference is not the same as it taking effect
Microsoft has to crawl the page to discover the setting. IndexNow can tell Bing a page changed, but Microsoft says it does not guarantee a recrawl date. A preference published on 28 February is not in force on 1 March. Set it early and check the AI performance report in Bing Webmaster Tools for the change.
What did Google announce, and why is its version different?
Google's post is shorter. It plans to support the same shared policy token in robots.txt in 2027, so that a website can express its AI training preference across multiple AI providers in a single step, and it describes the proposal as consistent with the EU Code of Practice for general-purpose AI model providers, which treats robots.txt as a primary way to express machine-readable preferences. Search Engine Roundtable reports OpenAI and Anthropic signing on to the same draft; we could not read OpenAI's own post, which refuses automated requests, so we are attributing that part rather than quoting it.
The difference is in one sentence. Google says that when a website uses the shared token to opt out of training, and Google-Extended is not explicitly specified, it plans to interpret that as if the publisher had disallowed Google-Extended. On its own that sounds like a convenience. It stops sounding like one when you read what Google-Extended controls. Google's crawler documentation, updated on 30 October, says the token manages whether crawled content may be used for training future Gemini models and for grounding, which it defines as providing content from the Google Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. The post then says the quiet part: websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
One line, two outcomes
Add User-Agent: __usage_ai-training with Disallow: / to your robots.txt and nothing else. On Bing, from February, your pages stop training Microsoft's models and keep appearing in Copilot's answers, because grounding is a separate switch that is still on. On Google, from 2027, the same line is read as a Google-Extended disallow, and that one token also covers grounding in the Gemini app. You asked not to be trained on and, on one of the two platforms, you also stopped being quoted. The fix is to say what you mean twice: keep the shared line, and add an explicit User-agent: Google-Extended group with Allow: / so Google has no gap to fill. Google's post only states what happens when Google-Extended is not specified; it has not published a precedence rule for the explicit case, so treat this as the documented path rather than a guarantee, and check it when the control goes live.
Two boundaries worth being precise about. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal, in Google's words, and neither announcement mentions AI Overviews or AI Mode. The control for those remains the Search generative AI setting in Search Console that we covered when it reached every site on 31 August. So after all of this a site on Google can be in four states at once: indexed in Search, shown or hidden in AI Overviews, trained on or not, and quoted or not in the Gemini app, each decided in a different place.
Should a store or B2B site refuse AI training at all?
For a publisher whose product is the words on the page, the training question is a real commercial decision and we are not going to pretend otherwise. For a merchant or a B2B company it is a different calculation, and an easier one. Your product pages exist to be found and repeated accurately. The data on them is already in feeds you hand to these same companies on purpose: Shopify Catalog, Merchant Center, the ChatGPT ads app. Refusing to let the public page train a model gains you very little that the feed has not already given away, and refusing grounding costs you the answer. Microsoft put a number-free version of that in its own post: reduced brand visibility in AI experiences. The 19 October study we covered found official sources in 84.4% of ChatGPT's fact answers; the assistants are going to your site to check, and a grounding block is a note on the door telling them not to.
What makes this worth an afternoon now rather than in February is who will actually set the line. Not you. Robots.txt is edited by SEO apps, agencies, theme updates and "block AI bots" plugins, and a shared token backed by Google, Microsoft, OpenAI and Anthropic is exactly the kind of thing that becomes a checkbox labelled Protect my content. Someone with good intentions will tick it. On Bing that costs you nothing. On Google, unless the explicit allow is already there, it costs you Gemini.
- Find out what you are already saying. View the source of a product page, a collection page and the home page and search for noarchive and nocache, in robots meta tags and in X-Robots-Tag response headers. On Shopify, check SEO apps and the theme's head snippet; on WordPress, the SEO plugin's advanced settings. Write down what you find and why it is there.
- Decide the two questions separately and in writing: may AI companies train on these pages, and may they quote them in answers. For most stores the honest answer is "we do not much care" and "yes, please". Record it so the next person who edits robots.txt knows.
- If you choose to refuse training, add the shared group and an explicit Google-Extended Allow in the same edit, and leave Bing's grounding preference at its default. Do not use a page-level ai-grounding:disallow unless you mean to leave the answer.
- Do not confuse any of this with crawl access. A Disallow for Bingbot or Googlebot still blocks everything, and the WAF and bot rules we wrote about in August still turn agents away before a preference is ever read.
- If noarchive is on your pages on purpose and you want to stay out of Copilot's answers, put 15 December and 1 March in the calendar: publish the page-level grounding preference first, then record the temporary one in Bing Webmaster Tools before the deadline.
- After 1 February 2027, check Bing Webmaster Tools' AI performance report and Search Console's generative AI data for a change in the weeks that follow, and compare it with the state you wrote down in step one.
Where this fits with what we have already covered
In August we showed that ChatGPT ignores noindex and no-store when it decides what to keep, which is a reminder that these preferences only bind the companies that have promised to read them. In September we covered the Search Console control that decides whether you appear in AI Overviews and AI Mode, which neither of this week's announcements touches. In August we also covered the CDN and firewall rules that turn AI crawlers away before any tag is read. This week adds the last layer: the explicit, machine-readable statement of what a page may be used for once the crawler is let in. It is the first time the two biggest search companies have agreed on the words, and the first time one of them has said in writing that being quoted and being trained on are different questions.
Where we fit
Our free AI Shopping Visibility Audit already reads your site the way these crawlers do: robots.txt, meta robots tags, X-Robots-Tag headers, CDN and bot rules, structured data and the plain text an answer is built from. Starting now, it also reports what your pages currently say about training and grounding, where the noarchive tags came from, which pages Copilot has been quietly skipping, and the exact robots.txt groups and header lines to publish so that February changes what you intended it to and nothing else. You get the findings as a short report with the lines ready to paste, and if an SEO app or theme is the thing writing the tags, we tell you which one and what to change in it.
Sources
- Microsoft Bing: Your content, your choice: clearer preferences for training and grounding (30 October 2026; the two preferences, their values, the syntax, the defaults, the visibility warning and every transition date)
- Microsoft Bing: New AI preferences FAQ (Bing Webmaster Tools help; the companion documentation named in the announcement)
- Microsoft Bing: Announcing new options for webmasters to control usage of their content in Bing Chat (22 September 2023; what noarchive and nocache have meant for AI answers and training until now)
- Google Search Central: Supporting a shared AI training control for websites (30 October 2026; the shared policy token, the 2027 timing, the Google-Extended interpretation and the Gemini grounding sentence)
- Google Search Central: Google's common crawlers (updated 30 October 2026; what Google-Extended controls, the early 2027 note and the Search inclusion and ranking statement)
- IETF: AI Preferences: training preference signal (the Internet-Draft both announcements link to)
- Search Engine Roundtable: Microsoft Bing separates AI training from AI search grounding with new controls (30 October 2026)
- Search Engine Roundtable: Google also adopts new AI training control (30 October 2026; the report that OpenAI and Anthropic are adopting the same draft)
- JTS Tech: What ChatGPT actually reads from your site (19 August 2026; noindex and no-store ignored)
- JTS Tech: Google now shows your AI impressions (1 September 2026; the Search Console control for AI Overviews and AI Mode)
- JTS Tech: Your bot protection is blocking AI shoppers (3 August 2026; the CDN and firewall layer)
- JTS Tech: The AI checks your own site now (19 October 2026; official sources in 84.4% of fact answers)
