Learn

Learn web crawling and scraping

A short, ordered course on how the web works for crawlers: HTTP, HTML and the DOM, selectors, rendering, sessions, and the boundary between crawling and scraping.

Six short lessons, in order. They cover the parts of the web a crawler actually touches, so the guides that follow make sense rather than feeling like recipes.

Last updated 2 November 2026

01What is web crawling? The crawl loop and its limitsWeb crawling is automated discovery and fetching of pages; scraping is extracting structured data from them. Learn the crawl loop, frontier, politeness, and when an API beats both.Updated 2 November 2026 · 5 min read

02HTTP for crawlers: status codes, headers, redirects, and conditional requestsA crawler's view of HTTP: methods, status classes, caching and conditional requests, redirects, compression, and the headers that change what you should do next.Updated 2 November 2026 · 4 min read

03HTML and the DOM: how to turn a page into a tree you can queryHTML is text; the DOM is the tree a parser builds from it. Learn elements, attributes, text nodes, parser differences, and the selectors that survive a redesign.Updated 2 November 2026 · 4 min read

04CSS selectors and XPath for scraping, with the pitfalls that break themLearn the CSS and XPath patterns that survive real pages: attributes, descendants, siblings, text matching, tables, and the selectors you should avoid.Updated 2 November 2026 · 4 min read

05Static vs JavaScript rendering: how to tell what a crawler actually receivesA server-rendered page and a client-rendered app look the same in a browser and completely different to a crawler. Learn how to tell, when to use a headless browser, and what it costs.Updated 2 November 2026 · 4 min read

06Sessions, cookies, and APIs: how state changes a crawlHTTP is stateless; cookies and tokens add state. Learn how sessions work, when to keep cookies, how to use an official API, and where authentication becomes a legal boundary.Updated 2 November 2026 · 4 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.