HTTP for crawlers: status codes, headers, redirects, and conditional requests

HTTP is the request-response protocol a crawler uses to fetch a URL. The parts that matter most are the method, the status code, the response headers, and the body. A crawler that reads only the body will retry the wrong failures, cache the wrong responses, and follow redirects it should have stopped at.

The request line and the method

A crawler almost always sends GET. Use HEAD only when you need the headers without the body and the server supports it; many servers implement HEAD inconsistently. POST belongs to APIs and forms, not to “crawling a page”. If a page requires a POST to show content, treat it as an API and read its documentation.

Two request headers are worth setting deliberately:

• User-Agent: a descriptive product token with a contact URL.
• Accept: what you can parse, for example text/html,application/xhtml+xml or application/json.

Do not set Accept-Encoding by hand in most clients; requests and httpx negotiate gzip/br and decode it for you.

Status codes: what to do next

The first digit tells you the class, and the class tells you the policy.

200 does not mean the page is the page you wanted. Soft 404s return 200 with an error page. Check the content, not just the status.

import requests

response = requests.get(
    "http://127.0.0.1:8765/missing",
    headers={"User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"},
    timeout=10,
)
print(response.status_code)
print(response.headers.get("content-type"))

Tested on 2 November 2026 against a local fixture that returns 404:

404
text/html; charset=utf-8

Headers a crawler should read

• Content-Type tells you how to parse the body. text/html, application/json, application/xml, and text/plain need different parsers.
• Content-Length is the declared body size. A missing or wrong length is a warning sign, not a reason to crash.
• ETag is an opaque validator for the representation. Send it back in If-None-Match to ask “has this changed?”.
• Last-Modified is a timestamp validator. Send it back in If-Modified-Since.
• Cache-Control says how long a response may be reused and by whom.
• Retry-After tells you when to retry after 429 or 503.
• Location is the target of a redirect.
• Set-Cookie is a state-setting instruction. Keep it only if your crawler needs a session.

Conditional requests: do not download what you already have

If the server gives you an ETag or Last-Modified, store it with the record. On the next crawl, send it back. A 304 Not Modified means you can reuse the stored body. This is the cheapest form of incremental crawling.

import requests

url = "http://127.0.0.1:8765/guides/http"
headers = {
    "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    "If-None-Match": '"fixture-v1"',
}

response = requests.get(url, headers=headers, timeout=10)
print(response.status_code)
print(response.headers.get("etag"))

Tested on 2 November 2026 against a fixture that returns 304 for that validator:

304
"fixture-v1"

A 304 has no body. Do not overwrite the stored record with an empty string; update the “last checked” timestamp instead.

Redirects

A redirect is not an error, but it is a decision. Follow 301/308 permanent redirects within a small hop limit (five is generous). Follow 302/303/307 temporary redirects if the target is still within the scope you intended. Record the final URL and the chain, so you can detect loops and update your canonical URL.

import requests

response = requests.get(
    "http://127.0.0.1:8765/old-guide",
    headers={"User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"},
    timeout=10,
    allow_redirects=True,
)

print(response.status_code)
print(response.url)
print([history.status_code for history in response.history])

Tested on 2 November 2026 against a fixture with a single 301:

200
http://127.0.0.1:8765/guides/http
[301]

If you set allow_redirects=False, you get the 301 and the Location header instead. That is useful when you are auditing redirects and do not want the client to hide the chain.

Cookies and sessions

HTTP is stateless; cookies add state. A Set-Cookie header asks the client to store a name/value pair and send it back on matching requests. requests.Session and httpx.Client manage a cookie jar for you.

Use a session when the site needs one: a locale preference, a consent state, an authenticated API, or a CSRF token paired with a form. Do not keep cookies you do not need; they are identifiers, and accumulating them across unrelated requests is a privacy and hygiene problem.

import requests

with requests.Session() as session:
    session.headers["User-Agent"] = "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"
    first = session.get("http://127.0.0.1:8765/session", timeout=10)
    second = session.get("http://127.0.0.1:8765/whoami", timeout=10)
    print(first.status_code, second.status_code, second.text.strip())

Tested on 2 November 2026 against a fixture that sets a session cookie:

200 200 session=fixture

Compression

Servers commonly compress responses with gzip, deflate, or Brotli. requests and httpx advertise supported encodings and decode the body automatically. If you use a lower-level client, you must decode it yourself. A body that starts with bytes that are not valid UTF-8 is often a compressed body being read as text.

Timeouts, retries, and the difference

A timeout is not a retry policy. Set a connect timeout and a read timeout, and handle them separately. A connect timeout usually means the host is unreachable; a read timeout may mean the server is slow or the response is large. Retry only failures that are safe to retry: connection errors, timeouts, 429, and most 5xx. Do not retry 400, 401, 403, or 404 without changing the request.

The retries and idempotency guide covers the policy, and the rate limiting guide covers the timing.

A crawler's HTTP checklist

• Send a descriptive User-Agent with a contact URL.
• Set connect and read timeouts.
• Check the status class before parsing.
• Read Content-Type and parse accordingly.
• Store ETag and Last-Modified; use conditional requests next time.
• Follow redirects within a small hop limit, and record the chain.
• Honour Retry-After on 429 and 503.
• Keep cookies only when the crawl needs a session.
• Do not retry client errors that will not change.

The next lesson, HTML and the DOM, starts where the response body ends: how to turn HTML into a tree you can query.