How to Reduce Google's Crawl Rate Safely on a SaaS Site in 2026

To reduce Google's crawl rate safely, return a 503 or 429 status code with a Retry-After header to Googlebot for no more than one to two days. Google slows crawling across the whole hostname when it sees these errors. Longer than that, and affected URLs can start dropping out of the index.

On October 6, 2026, Google added Retry-After details to its crawl rate documentation, according to the Google crawling documentation changelog. It is a small edit, but it settles a question SaaS engineering teams ask during every incident: can we slow Googlebot down without hurting rankings? The short answer is yes, for a short time, and only with the right status codes.

This guide is for B2B SaaS marketers and the engineers they work with. It covers when throttling is the right call, the exact responses to send, what never to send, and how to confirm crawling recovers afterward.

Why Google Changed Its Crawl Rate Guidance in October 2026

Google rewrote the emergency section of its reduce crawl rate documentation and added Retry-After examples. The header is defined in RFC 9110, the HTTP semantics standard. It accepts either a delay in seconds or an absolute date and time in UTC.

The core rule did not change. Google still says that when its crawlers see many 500, 503 or 429 responses, the crawl rate drops for the whole hostname. That includes URLs still returning content normally. Crawling rises again on its own once the error count falls.

What the update adds is a cleaner way to say when to come back. Before, a 503 told Googlebot "not now" with no hint about timing. A Retry-After value gives it a target. Google's examples use a 120-second delay and a fixed date, Wed, 21 Oct 2026 07:28:00 GMT.

When Should a SaaS Site Reduce Google's Crawl Rate?

Reduce Google's crawl rate only when Googlebot traffic is actually hurting your servers, such as during an outage, a database migration or a sudden spike that slows the app for paying users. Find the cause first. Google lists faceted navigation, sort and filter parameters and endless calendar URLs as the usual sources of runaway crawling.

Check your server access logs before touching any response codes. Google's own advice is to review recent logs, ideally with your hosting company, to confirm where the traffic is coming from. A spike that looks like Googlebot is not always Googlebot. Fake crawlers spoof the user agent string all the time, so verify the requests against Google's published IP ranges.

Load problems can also come from attackers rather than crawlers. The Cybersecurity and Infrastructure Security Agency (CISA), with the FBI and MS-ISAC, published a joint guide to distributed denial-of-service attacks in March 2024. It groups them into volumetric, protocol and application attacks. If your logs show one of those patterns, throttling Googlebot fixes nothing. That is an incident response problem.

Several situations are good reasons to slow Googlebot down. Here are the most common ones on SaaS sites:

How to Reduce Google's Crawl Rate With 503, 429 and Retry-After

The emergency method has four steps, and the order matters. Each one keeps the throttle narrow and short.

Here is what a correct response looks like on the wire:

HTTP/1.1 503 Service Unavailable
Retry-After: 7200

Keep the error page itself static and small. During an outage, the last thing your origin needs is a heavy React error page rendering thousands of times an hour.

What the Retry-After Header Does and Does Not Promise

Retry-After is a hint, not a contract. Google's documentation says crawlers can use it to decide when to retry. It does not say Googlebot will wait exactly that long, or return at exactly that moment. Treat the value as your best estimate.

The header also does not protect a URL indefinitely. If Googlebot sees the same 503 on the same URL "for multiple days," Google says that URL may be removed from the index. A Retry-After date three weeks out does not change that. Google gives no exact day count, which is why 48 hours is the practical limit.

At Cekpr, we read this October edit as a clarification, not a new feature. The useful part is the clear limit: 503 and 429 only, short windows only. For teams that run maintenance on a fixed weekly schedule, a dated Retry-After value is a tidy fit.

What Not to Do When You Slow Googlebot Down

Most crawl rate incidents that turn into ranking incidents come from the wrong fix, not the original problem. Google's guide to temporarily pausing a website lists the moves that remove pages from Search.

These responses do lasting damage. Avoid all of them during a slowdown:

The robots.txt trap is the one engineers hit most. A blanket Disallow: / feels like a polite "come back later." To Google, it reads as a permanent instruction. If you need finer control over which bots hit which paths, our robots.txt setup guide for AI crawlers covers per-agent rules that do not touch Googlebot's access.

The same caution applies to AI crawlers. If you also publish an llms.txt file, keep it serving normally during an incident. Our explainer on whether a SaaS site needs llms.txt covers what Google's own AI guidance says about that file.

Planning Crawl Slowdowns Into Maintenance Windows

The best crawl rate emergency is the one you planned. The National Institute of Standards and Technology (NIST) Contingency Planning Guide, SP 800-34 Rev. 1, frames outages around a business impact analysis: what a system does, how long it can be down and what has to happen first. That framing works for a marketing site too.

A simple runbook for a SaaS marketing site covers three things:

Write that runbook down before you need it. NIST's guide also stresses testing the plan, and a five-minute dry run on a staging host catches most mistakes. Common ones include a rule that matches every user agent or a missing header on cached responses.

For outages longer than two days, change approach. Google suggests serving a 200 placeholder page instead of a long run of 503s. A short notice page that stays online keeps URLs indexed far better than a week of errors.

How to Confirm Crawling Recovers After Throttling

Recovery is automatic, but you should still watch it. Google says crawling increases on its own once the error count falls. Open the Crawl Stats report in Search Console and check three lines over the following week: total crawl requests, the share of 5xx responses and average response time.

A healthy recovery shows the 5xx share falling to near zero within a day of the fix. Total requests should climb back toward the old baseline over several days. If total requests stay flat after a week, look for a leftover rule still serving 503 to some paths.

Two other checks help. Use URL Inspection on a few key product and pricing pages to confirm they still show as indexed. Then compare impressions in the Performance report against the prior week. If rankings dipped at the same moment a Google spam update rolled out, separate the two causes. Our backlink audit for the September 2026 spam update walks through that check.

Throttling also reaches past classic search. Pages Googlebot cannot reach for days are harder for AI answers to cite. Our analysis of where AI answers get their sources for B2B software shows how few citations point to a vendor's own site. Losing pages from the index makes that share smaller.

The Safe Default for SaaS Teams

The safe default is narrow and short. Send 503 or 429 with Retry-After, apply it only where the load is, and remove it within one to two days. Everything else in Google's guidance protects you from turning a server problem into an indexing problem.

Your next step is small. Write the CDN rule now, test it on staging and add the 48-hour stop to your incident checklist. Cekpr covers technical SEO, AI search and content for B2B software teams, and we will update this guide if Google changes the crawl rate documentation again.