- Decide what "stop" means first: block, challenge, rate-limit, or exclude from measurement. The four actions carry different costs, and most traffic does not need the heaviest one.
- Search and ad-platform crawlers must be allowed, and must receive exactly what real visitors receive. The server cost saved by blocking them is trivial next to dropping out of search or failing landing page checks.
- robots.txt is a request, not a lock. It works only on crawlers that choose to comply; anything that must be enforced belongs at the edge or in the application.
- Bot clicks on ads cannot be stopped before they are recorded. The practical goal is to find the sources that produce them and stop paying for those sources.
- Every blocking rule needs a false-positive measure. If blocks rise while the conversion rate of passing traffic falls, the rule is catching customers.
Define "stop" first: four actions, four costs
Many sites start by blocking outright and turn a reporting problem into a customer access problem. Before writing any rule, match the action to the type of traffic:
| Action | Suited to | Cost |
|---|---|---|
| Block | High-confidence abuse: known attack tooling signatures, clear credential-stuffing patterns | A false positive is a lost customer, and blocked people rarely report it |
| Challenge | Suspicious but uncertain requests, especially before high-value actions | Adds friction; some real users abandon at the challenge |
| Rate-limit | High-frequency requests from one source: search, login, APIs | Many real users behind one shared exit IP can be limited together |
| Exclude from measurement | Harmless automation that pollutes data: uptime probes, link previews, prefetching | Almost no access cost, but no reduction in server load |
A useful default order: exclude before rate-limiting, rate-limit before challenging, challenge before blocking. Reach for the heaviest action only when confidence is high and the cost of being wrong is acceptable.
Layer 0: the bots that must stay allowed
Every control starts from an allowlist. The following automated traffic should not be stopped, and stopping it causes more damage than the bots it was aimed at:
- Search engine crawlers. Block Googlebot or Bingbot and the site gradually disappears from search. Verify them with the reverse DNS method each search engine publishes, because spoofed crawler user agents are common.
- Ad-platform landing page fetches. These must receive exactly the page real visitors see. Blocking them produces destination-not-working disapprovals; serving them something different crosses the red line every platform prohibits.
- Services you depend on. Uptime monitors, payment webhooks, link previews from email and social platforms, partner integrations. First-draft rules often catch these, and the breakage is not always noticed quickly.
If crawl load is the worry, fix caching or adjust crawl-rate settings rather than blocking. AI crawlers and agent traffic are a separate population that needs its own decision; this article covers how to classify and monitor them.
What robots.txt can and cannot do
The usual answer to how to prevent bots from crawling a site is robots.txt, and it solves half the problem. robots.txt is a published request: compliant crawlers read it and follow it; scrapers, credential-stuffing scripts and click bots either never read it or ignore it. It is the right tool for telling search engines which paths are not worth crawling — internal search results, filter parameter combinations, carts — and the wrong tool for protecting anything. Restrictions that must be enforced belong in the layers below.
Layer 1: the edge (CDN and WAF)
The edge decides before a request reaches the origin, which makes it the highest-return layer. Most CDNs and hosting platforms include managed bot protection, and enabling it removes a large share of unsophisticated automation. The edge suits: known data-centre and hosting network traffic (challenge or log, depending on the business), clearly anomalous header and TLS characteristics, and sudden bursts from a single address. Edge bot detection covers which signals this layer reads.
The main risk at the edge is collateral damage. Blanket rules by network type catch real users on VPNs, corporate exits and some carrier networks. "Arrives via a proxy" and "is a bot" are different questions — proxy detection versus bot detection is worth reading before writing network-level rules. How commercial products differ in their verdict logic is covered in the bot detection systems comparison.
Layer 2: the application and APIs
The edge sees requests; the application sees behaviour. One address trying two hundred username and password pairs in a minute looks normal request by request, and only the application knows what the requests add up to. The main tool here is per-endpoint rate limiting:
- Login and password reset: limit per account and per source separately; challenge before locking once failures cross a threshold.
- Site search and filters: the favourite entry point for scrapers, and among the most expensive pages to serve.
- Price, stock and product APIs: public JSON endpoints are often easier to harvest than the pages themselves.
- Checkout and coupon validation: high-frequency attempts at codes or card numbers are classic automated abuse.
Set thresholds against the upper bound of real usage, not the average. Run each rule in log-only mode for one to two weeks, look at how many requests it would have caught and whose they were, then switch it to enforcement.
Layer 3: forms and sign-ups
The damage from junk sign-ups and fake leads is not on the server but downstream: sales time spent chasing fake leads, deliverability loss from mailing invalid addresses, and polluted bidding once those submissions are sent back to ad platforms as conversions. The combined toolkit at the form layer: hidden honeypot fields that people never see and scripts fill in, a minimum completion time (a long form finished in two seconds was not filled by a person), format and reachability checks on email and phone, and a challenge that appears only when risk is elevated. The step most often missed: do not report failed submissions back as conversions, or the ad platform will learn to find more of the same traffic.
Layer 4: paid clicks
Bot clicks on ads differ from every other layer in one fundamental way: the click is recorded and billed on the platform side, and nothing on the landing page changes that. The goal at this layer is therefore not to block but to find the source and stop paying for it. Platforms already filter a share of invalid clicks and credit them automatically — how Google Ads handles invalid clicks explains that part. On top of it, advertisers can:
- Compare clicks to engaged sessions per source. A placement, app or region whose clicks far exceed sessions on the landing page is the first signal.
- Exclude problem placements and IP ranges. In display and partner inventory, a small number of sources usually account for most invalid clicks.
- Tighten audience and geography settings. Loose targeting pulls budget toward the lowest-quality inventory.
- Keep visit-level records. When requesting additional credits from a platform, the only usable evidence is per-visit records reconciled against click reports.
Note the boundary at this layer: the landing page must be identical for ad-platform fetches and for real visitors. Stopping bot clicks never means showing the platform's checks a different page.
Layer 5: analytics and reporting
Some automation is not worth blocking but must be excluded from data, or bounce rate, conversion rate and attribution all skew. Turn on the analytics tool's built-in known-bot filtering; exclude your own probes, internal addresses and test traffic; and keep a separate segment for zero-engagement sessions instead of deleting them. How that segment's share moves over time is one of the best indicators of whether the whole stack is working.
Measuring the false positives
The most common failure in bot protection is not missing bots but blocking customers without anyone noticing. How bot protection blocks real customers has the full measurement method; the minimum version is this. For each rule, compare two weeks before and two weeks after on three numbers: the share of requests challenged or blocked per layer, the share of zero-engagement sessions, and the conversion rate of traffic that passed. The healthy result is the second falling while the third holds or rises. If blocks rise and conversion falls with them, the rule is catching customers. Check the most exposed segments separately: corporate networks, carrier gateways and in-app browsers.
The earlier basic steps for stopping bot traffic approaches the same problem from the ad-budget side and pairs well with this one.
Frequently asked questions
What is the quickest way to stop bot traffic on a website?
Turn on the managed bot protection your CDN or hosting provider already offers, add rate limits to login, search, sign-up and checkout endpoints, and exclude known bots in analytics. Those three steps handle most unsophisticated automation within a day. They do not handle everything, and each needs a false-positive check afterwards, but they are the highest-return starting point.
Does robots.txt stop bots from crawling my site?
Only bots that choose to obey it. robots.txt is a published request, not an access control. Search engines and reputable crawlers honour it; scrapers, credential-stuffing tools and click bots ignore it. Use it to guide well-behaved crawlers, and use edge or application rules for anything that has to be enforced.
Should I block Googlebot to reduce bot traffic?
No. Blocking Googlebot removes the site from Google Search over time, and blocking or serving different content to ad-platform crawlers causes landing page problems in paid campaigns. Verified search and ad-platform crawlers should receive exactly what real visitors receive. If crawl load is the concern, fix caching or crawl-rate settings rather than blocking.
How do I stop bots from clicking my ads?
Ad platforms already filter a share of invalid clicks and credit them automatically. On top of that, advertisers can exclude placements and IP ranges that produce clicks with no engagement, tighten audience and geography settings, and monitor click-to-session ratios per source. What cannot be done is stop a click from being recorded by the platform; the practical goal is to find the sources that generate it and stop paying for them.
Can blocking bots hurt real customers?
Yes. Aggressive rules catch real visitors on corporate networks, mobile carriers, privacy browsers and in-app browsers. Every blocking rule should be measured by what it removed: how many blocked sessions later showed human behaviour, and whether conversion rate on the affected segment dropped after the rule went live.
How do I know whether my bot controls are working?
Track three numbers over time: the share of requests challenged or blocked per layer, the share of analytics sessions with no engagement, and the conversion rate of traffic that passed. Controls are working when junk sessions fall while the conversion rate of passing traffic holds or rises. If blocks rise and conversions fall with them, the rules are catching customers.
ROAS365 keeps a visit-level record for every arrival: the source signals, the verdict, the reason behind it, and what that visit was actually served. Records export in raw form and reconcile against ad-platform click reports and downstream conversions — the data that the paid-click layer and false-positive measurement both depend on. To see how that fits an existing campaign structure, contact sales.