Skip to content

Guides

Crawling for RAG and LLM corpora: clean text, provenance, and permission

Build a retrieval corpus from public web pages: pick permitted sources, extract main text, chunk with provenance, deduplicate, evaluate retrieval, and refresh safely.

Last updated 3 min readBy CrawlerBros Editorial

Tested with Python 3.14.2, trafilatura 2.3.1 on 2 November 2026. Library versions and dates are part of the evidence, not decoration.

A RAG corpus is a curated, licensed, dated collection of clean text with provenance. The model is the last step, not the first. If the source is not permitted, the text is boilerplate, or the chunk has no URL and date, retrieval quality and legal risk both get worse.

This guide covers the pipeline from source selection to a refresher, with a tested main-text extractor.

1. Choose sources you may use

Before fetching, decide:

  • Is the content public and permitted by the terms?
  • Does the site’s robots.txt or AI-crawler policy allow the use?
  • Is there a licence (CC BY, open data, API terms) that covers storage and retrieval?
  • Does the content contain personal data, and do you have a basis to process it?
  • Do you need the full text, or is a summary/link enough?

A page being public does not make it a free training corpus. The AI crawler policy guide and copyright guide cover the boundaries.

2. Fetch politely

  • one request per URL, with a descriptive user agent and contact URL;
  • per-host rate limits;
  • robots.txt and Retry-After compliance;
  • a maximum page size and a timeout;
  • no login, paywall, CAPTCHA, or ban bypass;
  • a stop condition and a kill switch.

For a large corpus, fetch the HTML once, store the raw response briefly, and extract offline. That makes the extraction reproducible and avoids re-hitting the site when you change the parser.

3. Extract the main text

import trafilatura

html = open("rag-fixture.html", encoding="utf-8").read()
text = trafilatura.extract(
    html,
    include_comments=False,
    include_tables=True,
    include_links=False,
)
print(text)

Tested on 2 November 2026 with trafilatura 2.3.1 against a local article fixture:

RAG fixture
Crawling for retrieval needs clean text, a source URL, and a date. This paragraph exists to test extraction.
A second paragraph adds enough body text for the extractor to recognize the article.

The navigation and footer are gone. That is the point: a retrieval chunk should contain the article, not the cookie banner.

Extraction is not perfect. Tables, code blocks, and multi-column layouts can be mangled. Check a sample per source type, and keep the raw HTML until the extraction is validated.

4. Chunk with provenance

A chunk should carry:

  • the text;
  • the source URL;
  • the page title;
  • the published/updated date;
  • the section heading;
  • the fetch time;
  • a content hash;
  • the extractor version;
  • the licence/usage note.

A common starting point is 300–800 tokens with 10–20% overlap, split on headings and paragraph boundaries rather than fixed character counts. Do not split a table row or a code block in the middle if you can avoid it.

5. Deduplicate

Three layers:

  1. URL normalisation — strip tracking parameters and fragments.
  2. Exact content hash — the same article under a print or AMP URL.
  3. Near-duplicate detection — syndicated copies, boilerplate variants.

Deduplicate before embedding. Duplicate chunks waste context and make retrieval look better than it is.

6. Evaluate retrieval, not just the model

Build a small evaluation set: 30–100 real questions with the URL that answers each. Measure:

  • whether the correct source appears in the top-k;
  • whether the answer is supported by the chunk;
  • whether the citation is correct;
  • whether a stale chunk outranks a current one.

A corpus that cannot answer your evaluation questions is not ready, no matter how many tokens it has.

7. Refresh and delete

  • Re-crawl on a schedule matched to the source’s change rate.
  • Use ETag, Last-Modified, sitemap lastmod, and content hashes.
  • Keep a tombstone for deleted pages; do not keep serving a deleted article.
  • Honour deletion/erasure requests.
  • Re-run the evaluation after a refresh.

A RAG corpus without a deletion path is a liability, not an asset.

  • Source terms permit storage and retrieval.
  • robots.txt and AI-crawler policy checked.
  • Licence recorded per source.
  • Personal data minimised and retained for a set period.
  • Paywalls/logins/CAPTCHAs not bypassed.
  • Copyrighted text not republished beyond the licence.
  • Provenance stored with every chunk.
  • Deletion path tested.

Sources

Sources checked