Crawl-delay calculator: turn a politeness budget into a delay

This calculator converts a target request rate and a worker count into a per-worker delay. It also estimates how long a crawl of N pages will take. It is a planning tool, not a guarantee: real latency and server tolerance vary.

The formula

With C concurrent workers and a target of R requests per second, each worker should wait:

delay_per_worker = C / R

With one worker and 1 request/second, the delay is 1 second. With four workers and 1 request/second, each worker waits 4 seconds, so the aggregate is still about 1 request/second.

The estimated crawl time is:

time ≈ pages / R + latency × (pages / C)

The first term is the rate limit; the second is the processing time. The estimate assumes the workers are busy and the target responds consistently.

How to use it

1. Choose a target rate. For an unknown site, start at 1 request/second or lower.
2. Enter the number of pages and concurrent workers.
3. Read the per-worker delay and the estimated time.
4. Compare with the site’s Crawl-delay if it declares one. Treat the larger number as the floor.

Common Crawl-delay values

A Crawl-delay is not part of RFC 9309, but it is a clear request. If a site says 10 seconds, a four-worker crawler at 0.4 req/s is still faster than the request. Use one worker or honour the delay literally.

Limits

• The estimate ignores retries, failures, and variable latency.
• It assumes a single host. A multi-host crawl needs a per-host limiter, not one global delay.
• It does not know the site’s actual tolerance. Measure response times and error rates.
• It does not override a block or a terms restriction.

For the policy, read rate limiting and backoff. For scheduling, read scheduling and monitoring.

Sources

• RFC 9309
• Google Search Central — robots.txt