Back to the homepage
AI Search

How to Prepare Your SaaS robots.txt for Google's Shared AI Training Opt-Out

Google will support one robots.txt token for AI training preferences. Here is what B2B SaaS teams should check first.

Owen Mercer · AI Search and Analytics Writer
Start reading ↓
Desk
AI Search
Format
How-to
Reading time
7 min
Published
Nov 18, 2026

Google plans to support a shared robots.txt signal in 2027 that lets a site opt out of AI training across several providers in one step. The signal is a policy token, __usage_ai-training. It states a training preference, not a crawl rule. Audit your robots.txt now, then decide which parts of your SaaS site you want kept out of training.

This guide is for B2B SaaS marketers, founders and the developers who edit robots.txt. It explains what Google announced, what the token looks like, and what an opt-out does and does not change. cekpr covers AI search for software companies, and this is one of the rare changes where a one-line file edit can alter how your content is used.

What Did Google Announce About a Shared AI Training Control?

seo, optimization, search engine optimization, process

Google announced on October 30, 2026 that it plans to support a shared AI training control in robots.txt. A single policy token would let a site state its training preference across several AI providers. Google aims to support it, along with other providers, in 2027, and has not set a firm date.

The Search Central announcement says the work builds on the IETF's AIPREF working group and on Google-Extended, the training control Google has offered since 2023. Today a site that wants to opt out must name each company's crawler or token. The new signal aims to let a site cover several providers with one entry.

There is one rule for existing files. If a site uses the new token to opt out and does not mention Google-Extended, Google plans to treat that as if the site had disallowed Google-Extended. Company-specific controls keep working next to the shared one.

Specific rules should win over general ones, according to the draft. A site can opt out broadly with the shared token, then add an explicit Google-Extended rule to treat Google differently.

What Does the __usage_ai-training Token Look Like in robots.txt?

binary, binary code, binary system, http

The token is written as an ordinary robots.txt group. The draft behind it, an IETF Internet-Draft on the training preference signal, defines one token and adds no new directive names or grammar. Here is the example the draft gives:

text
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example expresses a preference against AI training for every path except /press-kit/, where it expresses a preference in favor. Allow and Disallow work as they do for any crawler, so you can set preferences by folder.

Three details matter for a SaaS site. Each one can cause a silent mistake:

  1. Wildcard groups such as User-Agent: * do not count as matching the token, so an old blanket rule does not opt you out.
  2. If your file has no group for the token, no preference has been expressed.
  3. Tokens must match exactly, apart from letter case, including the double underscore.

Because older parsers ignore groups they do not recognize, adding the token should not change how current crawlers read your file. At cekpr, we still recommend testing the file in staging before you publish it.

Does Opting Out of AI Training Remove You From AI Answers?

Opting out of training is not the same as opting out of AI answers. The IETF draft says the token is a statement about how content is used, not a directive for crawling. It also says the preference and crawler rules are independent, so your existing Googlebot rules keep working as before.

The open question is Gemini. Google's announcement says sites that specifically allow Google-Extended are eligible to appear in responses and get clicks via grounding in Gemini Apps. Grounding means the model checks live sources when it answers. A blanket opt-out read as a Google-Extended disallow could therefore affect that eligibility.

Google has not spelled out the effect on AI Mode or AI Overviews in that post. Treat those as unknown until documentation says otherwise. Do not assume an opt-out is risk free for AI visibility, and do not assume it is harmful. Check the facts on Google's Google-Extended documentation before you change production.

Measurement helps here. Our guide to measuring AI search visibility in Search Console shows how to baseline your numbers first. Then you can see whether a robots.txt change moves anything. You can also test whether your SaaS brand appears in Gemini and AI Mode answers before and after the edit.

Should a B2B SaaS Company Opt Out of AI Training?

newspapers, weekly, reading, information

A B2B SaaS company should decide by page type, not for the whole site. Marketing pages exist to be found and cited, so a blanket opt-out may work against you. Paid research, customer data and user-generated content have a stronger case for protection.

Use this table as a starting framework. It is a suggestion, not a rule, and your legal team should weigh in on contracts and licensed content.

Should a B2B SaaS Company Opt Out of AI Training?
Content typeSuggested stanceReason
Product and pricing pagesAllowYou want accurate descriptions in AI answers
Public docs and help centerAllowBuyers ask AI tools how-to questions
Blog and thought leadershipDecide per postCitation upside versus reuse concerns
Gated research and reportsOpt outThe report is the product
Community and user contentOpt out, check termsYou may not own the rights

There is a counterpoint. Some teams argue that opting out only removes your voice from the models people will use to research vendors. Others argue that training use brings no traffic back. Both views have merit, and no study that we can cite settles it.

Our post on tracking Gemini referral traffic in GA4 shows how to see whether Gemini sends visitors, which is a useful input for this decision.

How Does EU Policy Affect the Robots.txt Decision?

hammer, books, law, dish

EU policy treats robots.txt as the main way for publishers to state machine-readable preferences about AI training. Google's announcement says the proposal fits the EU Code of Practice for general-purpose AI models. The European Commission's guidance shows how seriously that channel is taken.

The Commission's consultation FAQ on AI and copyright explains that signatories of the Code commit to respect robots.txt. They also commit to comply with other appropriate machine-readable protocols for rights reservations. A clear robots.txt is therefore the first place a rights holder can state a position.

If you sell to European customers or publish original research there, this matters. The signal is a preference, not a legal contract, and it does not replace your terms of service. Ask counsel how your terms and your robots.txt should line up.

What Should You Do With Robots.txt This Month?

Do four things this month, none of which requires the token to be live. First, open your current robots.txt and list every AI-related group. Second, record which page types you would protect. Third, check that staging and production files match. Fourth, set a review date for early 2027.

Look for old rules that no longer match your intent. Teams often paste a block of AI crawler names from a blog post and forget it. Those lines may block crawlers whose output you actually want.

Also check where robots.txt lives. Each subdomain needs its own file. A docs subdomain or a help center hosted by a vendor may carry different rules from your main site, and robots.txt files apply per host.

Keep notes. When the feature goes live, you will want to know what you decided and why. A short entry that lists the date, the page types, and the reasoning is enough.

How Will You Know Whether the Token Is Working?

You will not see the token in your server logs. The IETF draft notes that a policy token need not appear in the User-Agent header of any outbound request. It is a label inside your file, not the name of a bot, so log analysis cannot confirm it. This surprises teams that rely on logs to check every robots.txt change.

Verification will come from three other places. First, a robots.txt parser can confirm that your file reads the way you intended. Second, each provider will need to publish its own documentation on how it honors the signal. Third, Google has said it will share updates on its progress, so watch the Search Central blog through 2027.

Until then, run a simple test file in staging. Include the token group, a wildcard group and a Googlebot group, then check that each one parses as expected. A wrong path in an Allow line is the most common error, and it can open a folder you meant to protect.

What Are the Common Mistakes With AI Training Opt-Outs?

The common mistakes are mixing up training and crawling, relying on a wildcard group, and blocking more than intended. Each one comes from the same habit: copying a snippet without reading what it covers. Slow down on this file, because one line affects the whole host.

  • Blocking Googlebot when the goal was only to opt out of AI training.
  • Expecting a User-Agent: * rule to apply to the new token.
  • Forgetting subdomains, which need their own robots.txt files.
  • Treating the signal as a contract instead of a stated preference.

A final point is about timing. The feature is planned, not live. Writing the token today expresses your preference to any consumer that already reads it, but Google has not committed to a date inside 2027.

The Bottom Line on the Shared AI Training Control

The shared control is a good idea with a long runway. It simplifies opt-outs, but it separates training from crawling and leaves some AI-answer effects unclear. Prepare now, and do not flip a switch because a headline says you can.

The recommendation is simple. Keep marketing and docs open, protect gated research and user content, and baseline your AI visibility before any change. Then edit robots.txt once, test it, and write down the date.

Will Google's shared AI training opt-out hide my site from Google Search?
The announcement describes a training preference, not a search control. The IETF draft says the token is not a crawling directive. Googlebot rules stay independent. Google has not detailed the effect on AI Mode or AI Overviews, so check its documentation before you rely on any assumption.
Can I use the __usage_ai-training token in robots.txt today?
You can write it today because older parsers ignore groups they do not recognize. Google plans to support it in 2027 with other AI providers, and has not set a firm date. Until a provider documents support, treat the line as a stated preference, not a guaranteed block.
Does User-Agent: * in robots.txt opt me out of AI training with the new token?
No. The draft says wildcard groups must not be read as matching the policy token. To express a preference, add a group that names __usage_ai-training directly. Keep your existing wildcard rules for crawlers, since crawling and training preferences are separate signals.
Which SaaS pages should I keep open to AI training?
Product, pricing and public documentation pages usually benefit from being found and cited, so many teams leave them open. Gated research, paid reports and user-generated content have a stronger case for opt-out. Review contracts and community terms first, and decide page type by page type.

Written by

Owen Mercer

Owen writes about how AI answer engines pick their sources and how to measure search visibility honestly. He is skeptical of any tactic that arrives without data.

More from Owen
Up next · Technical SEO How to Move a SaaS Site to a CDN Without Overloading Your Origin or Blocking Googlebot
The Thursday letter · free

join the club.

The week's best SEO and AI search posts, plus one thing to check in your own Search Console, every Thursday morning. Unsubscribe whenever.