Rate limiting and backoff: how to be a good crawler citizen

Rate limiting caps how many requests a crawler sends to a host in a period. Backoff increases the wait after a failure so a struggling server is not hammered. Both are politeness mechanisms first and reliability mechanisms second; a crawler that ignores them will be blocked, and it will deserve it.

Start with a per-host limit

A global limit is not enough. If you crawl ten hosts with a global 10 requests/second, one slow host can consume the budget and the others wait; worse, a single host can receive all ten requests in a burst.

A per-host limiter keeps a “next allowed time” for each host:

import time
from collections import defaultdict

class HostLimiter:
    def __init__(self, delay_seconds: float):
        self.delay = delay_seconds
        self.next_allowed = defaultdict(float)

    def wait(self, host: str) -> None:
        now = time.monotonic()
        wait_for = self.next_allowed[host] - now
        if wait_for > 0:
            time.sleep(wait_for)
        self.next_allowed[host] = time.monotonic() + self.delay

limiter = HostLimiter(delay_seconds=1.0)
limiter.wait("example.com")
print("request at", round(time.monotonic(), 3))
limiter.wait("example.com")
print("request at", round(time.monotonic(), 3))

Output (timing varies, but the gap is at least one second):

request at 12345.678
request at 12346.679

For concurrency, this simple limiter is not enough: multiple workers can pass the check before any of them updates the state. Use a lock around the check-and-set, or a token bucket with an atomic refill.

Token bucket

A token bucket allows short bursts up to a capacity, then settles to a steady rate. It is a better model for APIs that allow, say, 10 requests per second with a burst of 20.

import time

class TokenBucket:
    def __init__(self, rate_per_second: float, capacity: int):
        self.rate = rate_per_second
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()

    def take(self) -> None:
        now = time.monotonic()
        elapsed = now - self.updated
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.updated = now
        if self.tokens < 1:
            wait_for = (1 - self.tokens) / self.rate
            time.sleep(wait_for)
            self.tokens = 0
            self.updated = time.monotonic()
        else:
            self.tokens -= 1

bucket = TokenBucket(rate_per_second=2, capacity=4)
start = time.monotonic()
for _ in range(6):
    bucket.take()
print(round(time.monotonic() - start, 2))

Output:

1.0

Four requests consume the burst immediately; the next two wait for tokens at two per second, so the total is about one second.

Honour Retry-After

429 Too Many Requests and 503 Service Unavailable may carry Retry-After, either seconds or an HTTP date. Honour it. Sleeping less than the server asked is ignoring an explicit instruction.

import email.utils
import time
import requests

response = requests.get(
    "http://127.0.0.1:8765/",
    headers={"User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"},
    timeout=10,
)

if response.status_code in (429, 503):
    value = response.headers.get("Retry-After")
    if value:
        try:
            delay = int(value)
        except ValueError:
            delay = max(0, email.utils.parsedate_to_datetime(value).timestamp() - time.time())
        time.sleep(min(delay, 300))

Cap the wait. A server that asks for a day is telling you to stop, not to hold a worker for 86,400 seconds. Log the value and skip the host for that run.

Exponential backoff with jitter

Without jitter, many clients retry at the same intervals and create a thundering herd. Full jitter chooses a random wait between zero and the exponential ceiling:

import random
import time

def backoff_delay(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

for attempt in range(6):
    print(attempt, round(backoff_delay(attempt), 2))

Example output (random each run):

Retry only what can succeed on a retry: connection errors, timeouts, 429, and most 5xx. Do not retry 400, 401, 403, or 404 unchanged. A 403 is a decision, not a transient failure.

What rate should you choose?

There is no universal number. A defensible starting point:

One request per second per host for an unknown site.
One request every 5–10 seconds for a small site or a site with a Crawl-delay.
The API’s documented limit, minus a safety margin, for an API.
Zero if the terms prohibit automated access.

Measure the response time and error rate. If latency rises or 429/503 appears, reduce the rate. If the site publishes a Crawl-delay, treat it as a floor. The crawl-delay calculator converts a target rate and concurrency into a per-worker delay.

Logging and monitoring

Log at least:

host, timestamp, status, latency;
current rate limit and concurrency;
Retry-After values;
retry count per URL;
the number of URLs skipped because of robots.txt.

A crawl that is being blocked usually shows a rise in 403/429 and a drop in records per page before it fails outright. Alert on the trend, not just the crash.