Robots.txt for AI Crawlers: A Setup Guide for SaaS Sites

A SaaS robots.txt file for AI crawlers should separate search from training. Allow the crawlers that put your pages in AI search answers, such as Googlebot and OAI-SearchBot. Then decide separately whether to block training tokens like GPTBot and Google-Extended. Blocking Google-Extended doesn't remove you from Google Search, and that one detail trips up many teams.

Robots.txt used to be a set-and-forget file. A developer wrote it at launch, blocked the staging paths and nobody touched it again. AI crawlers changed that, because each vendor now ships several user agents with different jobs.

This guide is for SaaS marketers and founders who own their site's technical SEO, or who need to brief the engineer who does. It covers how robots.txt rules are actually parsed, which AI tokens do what and a sample file you can adapt. Cekpr covers technical SEO and AI search for B2B software teams, and robots.txt is the most common AI visibility mistake we find in audits.

How do crawlers read a robots.txt file?

Crawlers follow the Robots Exclusion Protocol, published by the Internet Engineering Task Force (IETF) as RFC 9309 in September 2022. A crawler finds the group that matches its user agent, case-insensitively. It then applies the most specific matching rule, meaning the longest path match, to decide whether a URL is allowed.

Three details in RFC 9309 matter for SaaS sites. First, crawlers must parse at least 500 kibibytes of the file, so a bloated file can get cut off. Second, crawlers shouldn't use a cached copy for more than 24 hours, so changes take effect within about a day.

Third, error codes change behavior. A 4xx response lets crawlers treat the file as missing and crawl everything. A 5xx response means crawlers must assume a complete disallow, which is how a misconfigured CDN can quietly stop all crawling.

Search crawlers versus training crawlers

The key decision is which AI uses you want to allow. Some crawlers fetch pages so an AI search product can cite them. Others gather content to train future models. A SaaS company usually wants the first and may not want the second.

The table below maps the main tokens from Google and OpenAI to their purpose.

Other AI vendors publish their own tokens. Check each vendor's crawler documentation before adding a rule, since names and purposes change.

Keep a short internal note listing every AI token in your file and why it is allowed or blocked. When a vendor renames a crawler or adds a new one, that note tells the next person what the original decision was.

What does Google-Extended actually control?

Google-Extended controls whether content Google crawls can be used to train Gemini models and to ground Gemini answers. Google's documentation on common crawlers says it doesn't affect a site's inclusion in Google Search. It also has no separate HTTP user agent, so it never shows up in your logs.

That last point confuses people reviewing server logs. Google crawls with its existing user agents and applies the Google-Extended rule as a usage control afterward. You can't confirm the rule from log data alone.

The bigger risk runs the other way. Some teams block Googlebot on sections they consider "AI sensitive," thinking it only affects AI answers. Google's guidance on AI features and your website is clear that AI Overviews and AI Mode use normal Search eligibility. Block Googlebot, and those pages leave classic results too.

How OpenAI's crawlers handle robots.txt

OpenAI's crawler documentation lists three agents, and each robots.txt setting works independently. OAI-SearchBot surfaces sites in ChatGPT search. GPTBot crawls content that may be used to train OpenAI's foundation models.

ChatGPT-User is different. It visits a page when a user asks ChatGPT a question that needs it, so robots.txt rules may not apply. OpenAI says to use OAI-SearchBot rules, not ChatGPT-User rules, to manage search appearance.

OpenAI says its search systems take about 24 hours to reflect a robots.txt change. It also publishes IP range files for each crawler. Your security team can use them to verify that a request claiming to be OAI-SearchBot really is.

A sample robots.txt for a SaaS site

The file below allows search crawlers, blocks training crawlers and keeps private app paths out of every index. Adjust the paths to match your own app and docs structure.

User-agent: *
Disallow: /app/
Disallow: /api/
Disallow: /checkout/
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xml

This file doesn't need an explicit group for Googlebot or OAI-SearchBot. Both fall back to the wildcard group, which allows everything except the private paths. Adding a named group for a crawler would replace the wildcard rules for that crawler, so repeat the private paths if you ever add one.

Whether to block training is a business decision, not a technical one. The U.S. Copyright Office's report on copyright and artificial intelligence covers generative AI training in Part 3, released in pre-publication form on May 9, 2025. Some SaaS companies block training to protect proprietary content. Others allow it so models learn their product names and category.

How to test and roll out robots.txt changes

Robots.txt mistakes are expensive and easy to make. The steps below catch most of them before they reach production.

Step three catches the failure nobody expects. A firewall rule that returns 503 to unknown bots can hit robots.txt itself. Under RFC 9309, a crawler that gets a 5xx has to treat your whole site as disallowed.

Common robots.txt mistakes on SaaS sites

Most robots.txt problems on SaaS sites come from a short list of habits. None of them look dangerous in a pull request. The list below covers the ones Cekpr's editors flag most often in technical reviews.

The noindex point deserves emphasis. Teams sometimes disallow a gated content page to keep it out of AI answers. Because the crawler can't fetch the page, it never sees the noindex tag, and the URL can stay in the index anyway.

Subdomains need their own audit. Each host serves its own robots.txt, so www, docs, help and the app subdomain each need a separate check.

Where robots.txt fits in your AI visibility work

Crawl access is the floor, not the ceiling. Once the right crawlers can reach your pages, the content has to be worth citing.

Robots.txt is also where the documented controls live, which is why we argued in our explainer on whether a SaaS site needs llms.txt that it deserves attention first. After a rule change, track the effect with the routine in our guide to measuring AI search visibility in Search Console.

One caution on diagnosis. A traffic drop after a robots.txt edit might be a coincidence with something else, such as a Google update. Our backlink audit after the September 2026 spam update covers how to rule that out using rollout dates.

The robots.txt setup most SaaS sites need

Allow search crawlers, decide on training crawlers as a business choice and keep private app paths blocked for everyone. Never block Googlebot to manage AI answers, because it removes you from Search entirely.

Open your live robots.txt today and compare it with the sample above. If you find a blanket block on an AI search crawler you actually want, fix it, test it and check your logs tomorrow.