Skip to content

Guides

Deduplication for crawlers: URL normalisation and content fingerprints

Stop crawling the same page twice. Normalise URLs, strip tracking parameters, hash content, detect near-duplicates, and enforce uniqueness in the database.

Last updated 3 min readBy CrawlerBros Editorial

Tested with Python 3.14.2 on 2 November 2026. Library versions and dates are part of the evidence, not decoration.

Deduplication removes repeated URLs and repeated content from a crawl. URL normalisation handles the same page under different URLs; content fingerprints handle the same content under different URLs. A serious crawler needs both.

Without deduplication, a site with a calendar, a sort control, and a session parameter can generate thousands of URLs for one page. The crawl gets slower, the dataset fills with duplicates, and the site gets more requests than it should.

Normalise the URL

from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit


def canonical(url: str) -> str:
    parts = urlsplit(url)
    query = [
        (key, value)
        for key, value in parse_qsl(parts.query, keep_blank_values=True)
        if not key.startswith("utm_")
    ]
    query.sort()
    path = parts.path.rstrip("/") or "/"
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), path, urlencode(query), ""))


for value in [
    "HTTP://Example.com/a/?utm_source=x&b=2&a=1#frag",
    "https://example.com/a?a=1&b=2",
    "https://example.com/a/",
]:
    print(value, "->", canonical(value))

Tested on 2 November 2026:

HTTP://Example.com/a/?utm_source=x&b=2&a=1#frag -> http://example.com/a?a=1&b=2
https://example.com/a?a=1&b=2 -> https://example.com/a?a=1&b=2
https://example.com/a/ -> https://example.com/a

Normalisation rules to decide deliberately:

  • Lowercase scheme and host. Paths are case-sensitive on most servers; do not lowercase them.
  • Remove the fragment. It is a client-side reference; the server never sees it.
  • Sort query parameters. ?a=1&b=2 and ?b=2&a=1 are usually the same page.
  • Strip tracking parameters. utm_*, fbclid, gclid, and similar.
  • Remove a trailing slash only if the site treats both as the same page. A redirect test is the safe way to decide.
  • Resolve default ports. :80 for http and :443 for https can be dropped.
  • Normalise percent-encoding. Decode unreserved characters; do not decode reserved ones.
  • Do not normalise away meaningful parameters. ?id=1 and ?id=01 may be different records.

Content fingerprints

Two URLs can serve the same content: a print view, a mobile view, a syndicated copy, a paginated article. Hash the extracted content, not the raw HTML (which may contain timestamps and CSRF tokens).

import hashlib
import json


def fingerprint(record: dict) -> str:
    payload = json.dumps(record, sort_keys=True, ensure_ascii=False).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()


record = {"sku": "A-1", "name": "Mechanical keyboard", "price": "$89.00"}
print(fingerprint(record)[:16])

Output:

1f7a2c9e0b6d4a83

For near-duplicate detection (the same article with a different byline, or a product page with a different review count), use shingling and MinHash/SimHash. The idea: split the text into overlapping word shingles, hash each shingle, and compare the set similarity. A Jaccard similarity above a threshold (for example, 0.9) is a likely duplicate.

Do not use near-duplicate detection to collapse records that differ in a field you care about. A price change is a real change, not a duplicate.

Enforce uniqueness in the database

A Python set is not enough once the crawl is distributed or resumed. Use a unique constraint:

CREATE TABLE urls (
  canonical_url TEXT PRIMARY KEY,
  first_seen TEXT NOT NULL,
  last_seen TEXT NOT NULL
);

CREATE TABLE records (
  record_id TEXT PRIMARY KEY,
  canonical_url TEXT NOT NULL,
  content_sha256 TEXT NOT NULL,
  payload TEXT NOT NULL,
  UNIQUE (canonical_url, content_sha256)
);

INSERT ... ON CONFLICT DO NOTHING makes the operation idempotent. A retry cannot create a second row.

Deduplicate across pages

A record can appear on page 1 and page 2 during an update. Deduplicate by a stable business key (SKU, post ID, canonical URL), not by the page it appeared on. If there is no stable key, use the content fingerprint and keep the newest observation.

What not to deduplicate

  • Different language versions. /en/ and /fr/ are not duplicates.
  • Different sizes/colours. They are separate products even if the description is shared.
  • A page and its API representation. Keep both if they serve different purposes; link them.
  • A redirect target and the redirecting URL. Keep the redirect relationship; do not collapse the historical URL out of your analytics.

Sources

Sources checked