Guides

Practical web crawling and scraping guides

Tested, vendor-neutral how-tos for Python, JavaScript, headless browsers, pagination, retries, sitemaps, incremental crawls, storage, and scheduling.

Deep, tested how-tos. Each guide states its library versions and test date, runs its code, and links to the primary documentation it relies on.

Last updated 2 November 2026

—Python web scraping with requests and BeautifulSoupBuild a small crawler in Python with requests and BeautifulSoup: frontier, deduplication, robots.txt, per-host delay, pagination, JSONL storage, and the failure checks that matter.Updated 2 November 2026 · 4 min read

—Scrapy tutorial: a production-shaped spider with robots.txt, throttling, and feedsBuild and run a Scrapy spider that obeys robots.txt, throttles itself, follows links, and writes JSON Lines. Includes the settings that matter and the mistakes that cause bans.Updated 2 November 2026 · 4 min read

—Playwright web scraping in JavaScript: waits, pagination, and browser hygieneUse Playwright with Node.js to scrape pages that need JavaScript. Tested examples for navigation, element waits, pagination, JSON responses, cleanup, and memory-safe concurrency.Updated 2 November 2026 · 4 min read

—Crawlee tutorial: a JavaScript crawler with a queue, retries, and storageBuild a Crawlee CheerioCrawler with request queues, same-domain link discovery, concurrency, retries, and dataset storage. Tested with Node.js 24 and Crawlee 3.Updated 2 November 2026 · 2 min read

—Headless browsers for scraping: when to use one, what it costs, and how to keep it stableA headless browser is a real browser without a window. Learn when it is necessary, how much memory it costs, how to pool and recycle it, and why it is not a permission bypass.Updated 2 November 2026 · 3 min read

—Pagination and infinite scroll: find the real next pageFollow next links, cursors, offsets, and infinite-scroll requests without guessing page numbers. Includes tested Python examples and the stop conditions that prevent loops.Updated 2 November 2026 · 3 min read

—Rate limiting and backoff: how to be a good crawler citizenImplement per-host rate limits, token buckets, exponential backoff with jitter, and Retry-After handling. Tested Python examples and the numbers that keep a crawl polite.Updated 2 November 2026 · 3 min read

—Retries and idempotency for crawlers: retry the right failuresRetry connection errors, timeouts, 429, and 5xx with bounded backoff. Do not retry 403 or 404. Includes a tested Python retry loop, idempotency keys, and checkpoints.Updated 2 November 2026 · 3 min read

—Sitemaps and feeds for crawlers: discovery without guessing URLsParse XML sitemaps and sitemap indexes, use lastmod honestly, read Atom/RSS feeds, and build a discovery loop that does not re-crawl the whole site.Updated 2 November 2026 · 3 min read

—Incremental crawling and change detection: stop re-downloading the webUse ETag, Last-Modified, sitemap lastmod, content hashes, and checkpoints to fetch only what changed. Includes tested conditional requests and a practical state schema.Updated 2 November 2026 · 3 min read

—Deduplication for crawlers: URL normalisation and content fingerprintsStop crawling the same page twice. Normalise URLs, strip tracking parameters, hash content, detect near-duplicates, and enforce uniqueness in the database.Updated 2 November 2026 · 3 min read

—Data cleaning and storage for scraped data: schemas, validation, JSONL, and databasesTurn raw scrape output into usable data: define a schema, validate required fields, store raw and normalised records, choose JSONL/Parquet/SQLite/Postgres, and set retention.Updated 2 November 2026 · 3 min read

—Scheduling and monitoring a crawler: cron, locks, metrics, and alertsRun crawls on a schedule without overlapping jobs, duplicate writes, or silent failures. Covers cron and systemd timers, locks, checkpoints, metrics, and alert thresholds.Updated 2 November 2026 · 3 min read

—Proxies and IP rotation for crawling: what they are, and when not to use themDatacenter, residential, mobile, and ISP proxies explained. Learn how proxies affect politeness and legality, how to configure one, and why rotation is not a permission bypass.Updated 2 November 2026 · 3 min read

—Finding XHR and GraphQL endpoints: scrape the data, not the pixelsUse the browser network panel to find the JSON or GraphQL request behind a page. Learn what to copy, how to replay it, and where authentication becomes a boundary.Updated 2 November 2026 · 3 min read

—API-first scraping: when the endpoint is better than the pageHow to find and use an official API before scraping HTML: docs, auth, pagination, rate limits, schema, and the cases where an API is not the right answer.Updated 2 November 2026 · 4 min read

—Selenium web scraping in Python: waits, pagination, and browser cleanupUse Selenium with headless Chromium to scrape a JavaScript-rendered page: explicit waits, element extraction, pagination, driver cleanup, and the limits of a browser.Updated 2 November 2026 · 3 min read

—Cheerio web scraping in Node.js: fast HTML parsing without a browserUse Cheerio with Node.js to parse server-rendered HTML, extract records, follow pagination, and know exactly when Cheerio is the wrong tool.Updated 2 November 2026 · 2 min read

—Scraping PDFs in Python: text extraction, tables, and scanned documentsExtract text from public PDFs with pypdf, handle multi-page documents, detect scanned pages, and know when you need OCR or a different tool.Updated 2 November 2026 · 3 min read

—News data for crawlers: RSS, Atom, and official APIs firstCollect news data from RSS/Atom feeds and official APIs, respect paywalls and robots, deduplicate stories, and keep a dated provenance record.Updated 2 November 2026 · 3 min read

—Bot protection and Cloudflare: why a crawler gets blocked, and what to doWhat WAFs, bot management, JS challenges, rate limits, and CAPTCHAs actually do, how to diagnose a block, and why evasion is not a scraping strategy.Updated 2 November 2026 · 4 min read

—Crawling for RAG and LLM corpora: clean text, provenance, and permissionBuild a retrieval corpus from public web pages: pick permitted sources, extract main text, chunk with provenance, deduplicate, evaluate retrieval, and refresh safely.Updated 2 November 2026 · 3 min read

—Async Python scraping with httpx: bounded concurrency without chaosUse httpx.AsyncClient with timeouts, connection limits, semaphores, and per-host rate limits. Tested async code and the cases where sync requests is still better.Updated 2 November 2026 · 2 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.