Incremental crawling and change detection: stop re-downloading the web

An incremental crawl fetches only pages that may have changed since the previous run. It uses HTTP validators (ETag, Last-Modified), sitemap lastmod values, content hashes, and stored checkpoints to avoid re-downloading the same bytes. It is the difference between a crawler that scales and one that re-reads the whole site every night.

Use HTTP validators first

If the server sends an ETag or Last-Modified, store it with the record. On the next request, send it back. A 304 Not Modified means the stored body is still current and you can skip parsing.

import requests

url = "http://127.0.0.1:8765/etag"
first = requests.get(url, timeout=10)
print(first.status_code, first.headers.get("etag"))

second = requests.get(
    url,
    headers={"If-None-Match": first.headers["etag"]},
    timeout=10,
)
print(second.status_code, second.headers.get("etag"), len(second.content))

Tested on 2 November 2026 against the local fixture:

200 "fixture-v1"
304 "fixture-v1" 0

A 304 has no body. Do not overwrite the stored record with an empty string; update last_checked and keep the old body.

Content hashes

Some servers do not provide validators, or the validator changes for reasons unrelated to the content. Hash the body and compare it with the stored hash:

import hashlib

def content_hash(body: bytes) -> str:
    return hashlib.sha256(body).hexdigest()

print(content_hash(b"<h1>Hello</h1>")[:16])

Output:

4d0e2a3d5a2c1f0b

Use a cryptographic hash for exact change detection, not for near-duplicate detection. A one-character change produces a different hash, which is what you want for “did this page change?” and not what you want for “is this a duplicate of another page?”.

A state schema that supports incremental crawls

CREATE TABLE pages (
  url TEXT PRIMARY KEY,
  etag TEXT,
  last_modified TEXT,
  content_sha256 TEXT,
  last_checked TEXT NOT NULL,
  last_changed TEXT,
  status INTEGER NOT NULL,
  body_path TEXT
);

On each run:

1. Fetch the URL with the stored ETag/Last-Modified.
2. On 304, update last_checked and stop.
3. On 200, hash the body. If the hash matches, update last_checked; if it differs, write the new body, update content_sha256 and last_changed, and re-parse.
4. On 404/410, mark the page as gone; do not keep serving stale data.
5. On 5xx/429, keep the old record and retry later with backoff.

Store the raw body for a short window (for example, seven days) so a parser bug can be fixed without re-crawling. After the window, keep the extracted record and the hash, not the HTML.

Sitemap lastmod is a hint, not a promise

A sitemap’s lastmod can prioritise the crawl, but it is not proof that the page changed. Sites often set it to the build time or the current date. Use it to order the queue, then confirm with an HTTP validator or a content hash.

The same applies to feeds. A feed entry’s updated timestamp is useful for discovery; it does not prove the page body changed.

Change detection output

When a page changes, record what changed, not just that it changed:

• a unified diff of the extracted fields;
• the old and new hash;
• the old and new lastmod;
• the timestamp of the observation.

A diff lets you alert on a price change or a schema change without storing two full copies of every page.

Scheduling the incremental run

Run frequency depends on how often the data changes and how much freshness matters:

Faster is not always better. A page that changes once a week does not need a hourly crawl; it needs a weekly crawl with a conditional request. Match the cadence to the change rate and the site’s tolerance.

Common mistakes

• Ignoring 304. Parsing an empty body and overwriting the record with nulls.
• Trusting lastmod. Re-crawling everything because the sitemap says today.
• Hashing the rendered page. Dynamic timestamps, ads, and CSRF tokens change on every load; hash the content you extract, not the whole DOM.
• No tombstone. A page that returns 404 stays in the dataset forever.
• No checkpoint. A crash at 90% starts over.

Sources

• RFC 9111 — HTTP Caching
• RFC 9110 — HTTP Semantics
• sitemaps.org — lastmod