Retries and idempotency for crawlers: retry the right failures

A retry is a second attempt at a request that failed for a transient reason. Idempotency means repeating the request does not create a second side effect. A crawler should retry only failures that can succeed on a second attempt, with a bounded number of attempts and a backoff between them.

The most common mistake is retrying everything. A 403 will not become a 200 because you asked again; a 404 is a missing page; a 400 means the request is malformed. Retrying those wastes time, increases the load on the site, and can turn a temporary block into a permanent one.

What to retry

A tested retry loop

import random
import time

import requests


def retry_delay(attempt: int, response: requests.Response | None = None) -> float:
    if response is not None:
        value = response.headers.get("Retry-After")
        if value:
            try:
                return min(float(value), 30.0)
            except ValueError:
                return 0.0
    return random.uniform(0, min(30.0, 2 ** attempt))


session = requests.Session()
session.headers["User-Agent"] = "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"

for attempt in range(5):
    response = session.get("http://127.0.0.1:8765/flaky", timeout=5)
    print("attempt", attempt + 1, "status", response.status_code)
    if response.status_code not in (429, 500, 502, 503, 504):
        break
    delay = retry_delay(attempt, response)
    print("waiting", round(delay, 2))
    time.sleep(delay)

Tested on 2 November 2026 against a fixture that returns two 503s and then 200:

attempt 1 status 503
waiting 0.0
attempt 2 status 503
waiting 0.0
attempt 3 status 200

Real servers rarely send Retry-After: 0. With a real Retry-After: 10, the loop waits ten seconds before the next attempt. The cap stops a malicious or misconfigured header from parking a worker for hours.

Bound the attempts

A retry loop without a maximum can run forever. Set three limits:

• Attempts per URL (three to five is typical).
• Total time per URL (for example, two minutes).
• Consecutive failures per host (for example, ten; then pause the host and alert).

When a host crosses the threshold, stop sending requests to it. Continuing is not persistence; it is a self-inflicted block.

Idempotency

GET is defined as safe and idempotent: repeating it should not change server state. Most crawling is GET, so retrying is usually safe. POST is different: a retry can create a duplicate order, comment, or job.

If you must retry a POST, use an idempotency key supplied by the API. The key is a unique string per logical operation; the server returns the first result for a repeated key instead of creating a second record.

import uuid
import requests

key = str(uuid.uuid4())
response = requests.post(
    "https://api.example.com/v1/jobs",
    headers={
        "Authorization": "Bearer <token>",
        "Idempotency-Key": key,
        "Content-Type": "application/json",
    },
    json={"url": "https://example.com"},
    timeout=20,
)

This is illustrative; the header name and behaviour are defined by the API. Read the documentation. If the API does not support idempotency keys, do not blind-retry a POST; check the resource’s state first.

Checkpoints

A long crawl should be resumable. Store:

• the frontier (URLs not yet fetched);
• the set of fetched URLs (or a hash);
• the last successful page/offset/cursor;
• the run ID and start time.

On restart, load the checkpoint and continue. Without one, a crash at 90% means starting over. With one, the next run picks up where the last left off.

For a single-host crawl, a SQLite table is enough:

CREATE TABLE IF NOT EXISTS frontier (
  url TEXT PRIMARY KEY,
  status TEXT NOT NULL DEFAULT 'pending',
  attempts INTEGER NOT NULL DEFAULT 0,
  updated_at TEXT NOT NULL
);

Mark a URL done only after the record is written, not after the response is received. Otherwise a crash between fetch and write loses the record.

Retries and rate limits interact

A retry is still a request. It must go through the same per-host limiter. A retry loop that bypasses the limiter will increase the request rate exactly when the server is struggling.

Order of operations:

1. Acquire a rate-limit slot.
2. Send the request.
3. On a retryable failure, release the slot and schedule the retry after the backoff.
4. On success, release the slot and process the response.

Sources

• RFC 9110 — HTTP Semantics
• Google Cloud — Retry strategy
• Exponential Backoff and Jitter