Don't ban the Googlebot

A catch-all edge that feeds a deception dashboard and CrowdSec is a good place to notice a category error. The rule that gets proposed is almost always the same: block the scanners by User-Agent. It takes a minute to write, it never fires wrongly on your own browser, and it is close to worthless.

The client chooses the string

A User-Agent header is text the client writes. Any scanner can send Googlebot, and the ones that matter usually do, because impersonating a crawler is a cheap way past naive rules. So a User-Agent deny rule filters on a field controlled by the party you are filtering: it blocks the crawlers that identify themselves, and repels nobody who will not simply pick another string.

Which means the block lands on the traffic that pays for your site, not on the traffic you were worried about.

What can be decided, and by whom

There is one fact about a crawler that the client cannot forge: the address it connects from. The publishers document the ranges they originate from, so this fleet reads those documents instead of guessing.

1
2
3
4
5
6
7
8
sources = {
  googlebot = {
    url = "https://developers.google.com/search/apis/ipranges/googlebot.json";
    hash = "sha256-n17GcnxmJ4244sglbrP0VzILCXkWbiBJ/ka6nmUOgu8=";
  };
  googleSpecialCrawlers = { url = "https://developers.google.com/search/apis/ipranges/special-crawlers.json"; ... };
  bingbot = { url = "https://www.bing.com/toolbox/bingbot.json"; ... };
};

hosts/common/global/crawler-ranges.nix fetches three manifests and turns each prefixes list into CIDRs. The ranges are not copied into the repository. They are fetched at evaluation time and pinned by SRI hash, and the header says exactly why: a publisher that silently edits its ranges becomes a hash mismatch — a build failure a human reviews — instead of a silent widening, or a silent narrowing, of the fleet’s own defence.

A security control you cannot diff is a control you cannot review. A list of ranges that changes underneath you is not data you own; it is a number you hope is right. Pinning the bytes makes the change visible. It also makes the fetch a fixed-output derivation, so the URL is contacted only on a cold cache or a deliberate hash bump, not on every rebuild.

Fail closed, and where rDNS belongs

Two details turn the module from a list into a control.

If a manifest stops having the documented shape, evaluation aborts with the URL that changed, so the failure reads review this publisher’s new format rather than defend with an empty list. And every derived entry must match [0-9a-fA-F:.]+/[0-9]{1,3}, because CrowdSec parses whitelist entries with netip.ParsePrefix: a bare address makes the whole postoverflow file fail to compile and the agent refuses to start, which took this fleet down once. Catching it at evaluation means the error names the publisher instead of a truncated store hash.

Reverse DNS is deliberately not required on top. These are the publishers’ own ranges, and at BGP level nobody else originates from them, so a forward-confirmed PTR adds no assurance here — it would only lose the whitelist whenever reverse DNS is slow or missing for a legitimate crawler. The PTR check belongs with the publishers who document no ranges at all; CrowdSec’s hub seo-bots-whitelist matches crawlers by PTR suffix, which is the right check for them.

The consumer is a CrowdSec whitelist of its own, fleet/whitelist-verified-crawler-ranges, so a verified crawler never becomes a decision, whatever a scenario says. It is a separate whitelist rather than extra entries on the fleet’s own-infrastructure list because fleet.crowdsec.whitelistCidrs is a listOf option with a non-empty default: assigning it from a second module replaces that default outright and silently drops the mesh range. Every whitelist is evaluated independently, so an extra entry is equivalent and cannot clobber the first.

The asymmetry, and the limits

The cost of a false block is not symmetric. A blocked malicious client changes its User-Agent and is back in a second. A blocked search crawler is invisible to you — no error page, no alert, no mail — and expensive to detect, because the only symptom is traffic that never arrives.

Two limits are worth stating rather than hiding. An address inside a publisher’s range is not a personality: range allow-listing cannot tell two clients behind the same address apart, so it is a statement about origin, not intent. And an allow-list is a standing exemption, so it needs an owner and a review. The procedure attached to this fleet’s allowlist pull requests is exactly that discipline: the acceptable diff is one appended line, and anything unidentifiable escalates to a human instead of being merged.

What you can check

• The module, in full. nix-config/hosts/common/global/crawler-ranges.nix. The three hash = values and the throw on an unexpected manifest shape are the whole argument in code.
• That the ranges are the publishers’ own. Each url is a manifest you can fetch yourself; the hash only makes a change visible.
• That the whitelist is separate. Search the module for s01Whitelist; the entry is named fleet/whitelist-verified-crawler-ranges, not the own-infrastructure list.

The practical rule: filter by behaviour and by verifiable origin, keep the allow-list small, reviewed and diffable, and never ship a User-Agent deny rule as your security story. For the rest of the defence — the bans, the log line, the dashboard — see the logging stack and how the fleet is configured.