- Robots.txt only works on crawlers that choose to honor it. Googlebot, Bingbot and mainstream SEO tools do; scrapers and disguised traffic do not. It is a declaration, not a lock.
- A robots.txt that allows only Googlebot also shuts out Bing and other search engines, and can catch ad platform verification crawlers. List the crawlers you actually need before writing it.
- Real enforcement relies on verification: reverse plus forward DNS, or a match against Google's published IP ranges, to confirm a request claiming to be Googlebot really comes from Google. The user-agent string alone proves nothing.
- Access control and content consistency are separate questions. You may refuse access to some crawlers, but the crawlers you admit and real people must receive the same content; otherwise the setup falls within cloaking policy.
Why people want to block everything except Google
The request usually comes from three places. The first is server load: SEO tool crawlers such as AhrefsBot and SemrushBot, along with assorted scrapers, fetch aggressively enough to take a visible share of a small site's bandwidth and CPU. The second is content protection: owners do not want articles, prices or product data copied in bulk. The third, more recent, is AI crawlers: a growing number fetch pages for model training or retrieval, and many sites want to choose whether to take part. For identifying and monitoring that category, see our guide to AI crawler and agent traffic monitoring.
What the three have in common is that they are all about deciding who gets access, not what each visitor is shown. Keeping that distinction clear keeps every layer below on the right side of the line.
Three layers, each with its own job
| Layer | What it controls | Enforced? | Spoofable? |
|---|---|---|---|
| robots.txt | Whether a crawler should fetch certain paths | No, relies on the crawler's cooperation | Simply ignored by bad actors |
| noindex (meta tag or X-Robots-Tag) | Whether a fetched page enters the search index | Honored by search engines | Not about access at all |
| Server or edge rules | Whether a request gets a response, and how fast | Yes | Hard to spoof when paired with verification |
Layer one: what robots.txt can and cannot do
The usual “Google only” file gives Googlebot its own group that allows everything, then gives the wildcard * a group that disallows the whole site. Under the robots.txt rules, a crawler follows the group whose user-agent line matches it most specifically, so Googlebot reads its own allow group and every other compliant crawler falls into the * group.
Several details are easy to miss:
- Other search engines are blocked too. Bingbot, Applebot, Yandex and others fall into the
*group. If you want their organic traffic, give each one its own allow group. - Google's ad-checking crawlers need separate thought. Google documents that its AdsBot crawlers ignore the
*group and only follow rules that name them explicitly. A wildcard disallow therefore does not touch them, but naming and disallowing AdsBot directly interferes with landing page quality checks. - AI training uses separate control tokens. Google-Extended, for example, is a product token in robots.txt that declares whether content may be used for Google's generative model training. It is not a separate crawler and does not change Googlebot's search crawling.
- Disallowing a crawl is not the same as staying out of the index. A disallowed URL with external links can still appear in results without a snippet.
Most importantly, robots.txt has no force over visitors that choose to ignore it. It works like a sign on the door; real scrapers and automated traffic walk straight past it.
Layer two: noindex governs indexing, not crawling
Many searches for “how to block Googlebot” really mean “I do not want certain pages in search results.” For that goal the right tool is noindex, either a robots meta tag in the page or an X-Robots-Tag response header, rather than a robots.txt disallow. The reason is simple: a search engine has to be able to fetch the page to read the noindex instruction. Disallow the crawl as well and the noindex is never seen.
A common split: admin areas, staging environments and internal search result pages get noindex or a login wall; large sets of parameter URLs that waste crawl resources are handled in robots.txt; and pages that must leave the index urgently get a temporary removal in Search Console first, followed by a permanent noindex.
If the goal is only to ease Googlebot's load, there is no need to block it. Google states that it temporarily slows crawling when it receives 500, 503 or 429 responses. Blocking Googlebot long-term costs the pages their organic visibility over time.
Layer three: verifying that “Googlebot” really is Google
Once you need real blocking, you have to face one fact: the user-agent header is free text, and a meaningful share of requests claiming to be Googlebot do not come from Google. A rule that reads “allow anything whose user agent contains Googlebot” opens the door to every impersonator.
Google offers two verification methods:
- Reverse then forward DNS. Run a reverse DNS lookup on the requesting IP and confirm the hostname belongs to googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it resolves back to the same IP. Only both steps together count as verified.
- Match published IP ranges. Google publishes JSON files listing the IP ranges for its common crawlers, its special-purpose crawlers such as AdsBot, and its user-triggered fetchers. Pulling them on a schedule and matching at the edge scales better than a DNS lookup per request.
Bing and other major engines offer similar verification. Verified crawlers take the allow path; requests that claim to be a search crawler but fail verification can be treated as impersonation. This check complements general bot detection rather than replacing it; see proxy detection versus bot detection for how the two differ, and edge bot detection for running checks close to the visitor.
Block or rate-limit?
For clearly identified crawlers that honor robots.txt, such as SEO tools, a declaration in robots.txt is usually enough. For unidentified requests with abnormal fetch rates, rate limiting is often safer than an outright block: a 429 with a Retry-After header protects the server without shutting out real people through a false positive. For the wider picture, see layered controls for bot traffic on a website.
Why ad landing pages need extra care
Sites that run ads carry an extra risk. Ad platforms fetch landing pages with their own crawlers to confirm the page is reachable and matches the ad; Google's AdsBot and Meta's facebookexternalhit are examples. If a protection rule treats them as ordinary bots and blocks them, the usual outcome is a disapproval along the lines of “destination not accessible.” The page-level requirements are covered in Google Ads landing page requirements.
The boundary matters even more. Once these crawlers are admitted, they must see the same page real visitors see. Deciding whether a visitor gets in based on who they are is access control. Deciding which version they see based on who they are falls within the cloaking policies of every major platform. The line between the two is covered in more depth in cloaking versus compliant personalization.
Pre-launch checklist
- List the crawlers you need: which search engines, which ad platform verification crawlers, and whether you accept AI crawlers.
- Give every needed crawler an explicit allow group in robots.txt, and do not name any ad verification crawler in a disallow rule.
- Use noindex for pages that should stay out of search, without also disallowing them in robots.txt.
- Base server or edge rules on DNS or published-IP verification, not on the user-agent string alone.
- Prefer rate limits (429 plus Retry-After) for abnormal request rates, and review logs regularly for false positives.
- Confirm that every admitted visitor, crawler or human, receives the same page content.
Frequently asked questions
How do I allow only Google in robots.txt?
Give Googlebot a group that allows the whole site and give the wildcard * a group that disallows it. Compliant crawlers follow the group that matches their name most specifically. This also shuts out Bing and other engines, and only affects crawlers that honor robots.txt.
Does robots.txt stop scrapers and malicious bots?
No. Robots.txt is a voluntary protocol and visitors that choose to ignore it simply do. Real blocking has to be enforced at the server or edge with identity verification, behavioral signals and rate limits.
How can I verify a visitor is really Googlebot?
Run a reverse DNS lookup on the IP and confirm the hostname is under googlebot.com, google.com or googleusercontent.com, then a forward lookup to confirm it resolves back to the same IP. Alternatively, match the IP against the crawler ranges Google publishes.
Should I block Googlebot to keep a page out of Google Search?
No. Use noindex and let Google crawl the page so it can read the instruction. A robots.txt disallow alone can still leave the URL in results, without a snippet, if other sites link to it.
Will blocking SEO tool crawlers affect my rankings?
Not directly; those tools are not search engines. The cost is that your site's data in those tools, such as backlink and keyword estimates, becomes incomplete, which matters if you or your partners rely on it.
One URL, the same page for every visitor
Same landing-page URL for every visitor — language and region adaptation in-page, in-page A/B testing, and invalid-traffic filtering. No cloaking, no sneaky redirects.