CrawlerLimiter

Per-crawler "one active resource at a time" limiter.

Each known crawler User-Agent gets a single slot. While a slot is held, requests from that crawler for the same resource key (refresh the slot) and requests from that crawler for paths with no key (homepage, static assets, etc.) are allowed; requests for a different resource key are denied with 429 Too Many Requests until the slot's last access is older than hold.

The "resource key" is whatever you compute from the request — typically a coarse grouping that maps to an expensive backend operation. Examples:

• a Maven groupId/artifactId/version triple, when each new triple triggers a fresh archive download + extraction
• a tenant ID, when each new tenant warms an isolated cache
• a date bucket, when each bucket spans a separate database shard

Crawlers that don't legitimately need to walk the whole keyspace in parallel (Googlebot et al.) will simply move on to a different page from the same key while their slot is held; well-behaved crawlers absorb the limit invisibly.