Web crawling and scraping glossary

Each entry starts with a one-sentence definition, then the detail that matters in practice. Links go to the lesson or guide that covers the term in depth.

Web crawler

A web crawler is a program that discovers URLs, fetches them over HTTP, and follows links to find more. The crawler owns the frontier, scheduling, politeness, and storage; it does not necessarily extract structured fields. See how web crawling works.

Web scraping

Web scraping is the extraction of structured data from pages a crawler or fetcher has retrieved. The scraper turns HTML, JSON, or a rendered DOM into fields; the crawler decides what to fetch and when. The two are often one program, but they are different jobs.

Crawl budget

Crawl budget is the amount of crawling a search engine is willing to spend on a host in a period, shaped by crawl capacity and host load. It is a search-engine concept, not a limit you set in your own crawler. For your own crawler, the relevant limit is the site’s capacity and your politeness policy.

URL frontier

The URL frontier is the queue of URLs a crawler has discovered but not yet fetched, usually prioritised and de-duplicated. A frontier with no deduplication will crawl the same page repeatedly; one with no priority will crawl low-value pages before the pages that matter.

Robots.txt

robots.txt is a text file at the root of a site that tells crawlers which paths they may request. The matching rules are defined in RFC 9309; the file is advisory under the protocol but can be evidence of a site’s expectations in other contexts.

User agent

A user agent is the string a client sends to identify itself, usually a product token, version, and contact URL. It is self-declared, not authenticated. A crawler should use a descriptive token and a real contact URL rather than impersonating a browser.

Crawl-delay

Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait a number of seconds between requests. It is not part of RFC 9309, but many crawlers read it; a crawler that ignores it should still enforce its own rate limit. See the crawl-delay calculator.

Sitemap

A sitemap is an XML file that lists URLs a site wants crawled, optionally with a last-modified date. A sitemap index points to other sitemaps. It is a discovery aid, not a guarantee of indexing; see sitemaps and feeds.

Canonical URL

A canonical URL is the preferred URL for a piece of content when several URLs serve it. A <link rel="canonical"> declares the preference; a crawler should still deduplicate by content and URL normalisation, because a canonical tag is a hint, not a redirect.

HTTP status code

An HTTP status code is the three-digit result a server returns for a request, grouped into 1xx, 2xx, 3xx, 4xx, and 5xx. 200 means success, 301/308 permanent redirect, 404 not found, 429 too many requests, and 503 temporarily unavailable. Retry policy depends on the class; see HTTP for crawlers.

Rate limiting

Rate limiting is the practice of capping how many requests a client sends per unit of time, either for politeness or to stay within a server’s published limits. Implement it per host, not per process, and make it visible in logs. See rate limiting and backoff.

Exponential backoff

Exponential backoff is a retry strategy that multiplies the wait between attempts, usually with random jitter, so a failing service is not hammered by synchronized clients. A common sequence is 1 s, 2 s, 4 s, 8 s, capped, with full jitter.

Retry-After

Retry-After is an HTTP response header that tells a client when it may retry, either as a number of seconds or an HTTP date. On 429 and 503, honour it; retrying sooner is ignoring the server’s explicit instruction.

Headless browser

A headless browser is a browser engine driven without a visible window, used when a page needs JavaScript to produce the content or when you need real browser behaviour. It is expensive: memory, CPU, and startup time are all higher than an HTTP client. See headless browsers.

DOM

The DOM (Document Object Model) is the tree of nodes a browser builds from HTML, which scripts can query and modify. A static HTTP client sees the HTML source; a browser sees the DOM after parsing and JavaScript execution. The difference explains many “the selector works in DevTools but not in my scraper” bugs.

CSS selector

A CSS selector is a pattern that matches elements in a document, such as article.card h2 or a[href^="/docs"]. It is concise and well supported; use it for stable attributes and structure, not for generated class names that change on every deploy.

XPath

XPath is a query language for selecting nodes in an XML or HTML tree, with axes, predicates, and functions. It is more expressive than CSS for “the table cell after the heading that says Price” problems, but less readable for simple matches. See CSS selectors and XPath.

Pagination

Pagination is the split of a result set across multiple pages or requests, using page numbers, cursors, offsets, or “next” links. A crawler must follow the real pagination mechanism, not guess ?page=2, and must stop on a repeated page or a missing cursor.

Infinite scroll

Infinite scroll is a UI pattern that loads more results when the viewport nears the bottom, usually through an XHR or GraphQL request. The underlying request is often directly callable; find it in the network panel before reaching for a browser. See pagination and infinite scroll.

XHR

XHR (XMLHttpRequest) is the older browser API for background HTTP requests; the term is often used loosely for any asynchronous request a page makes. Modern pages more often use fetch, but the network panel shows both. The response is usually JSON, which is easier to parse than the rendered HTML.

GraphQL

GraphQL is a query language and server runtime for APIs in which the client asks for specific fields. A GraphQL request is usually a POST with a query document and variables; pagination is defined by the server’s schema, often with cursors. See finding XHR and GraphQL endpoints.

API

An API (Application Programming Interface) is a documented contract for requesting data or actions from a service. For crawling, an official API is usually preferable to scraping: it is more stable, has explicit limits, and states what is permitted. Start with the API before parsing HTML.

Session cookie

A session cookie is a cookie that ties requests to a server-side session, typically set after login. A crawler that needs a session must store and send cookies consistently; a crawler that does not need one should discard them rather than accumulate tracking identifiers.

Authentication

Authentication is proving who a client is, usually with a password, token, key, or certificate. Authentication is not authorisation: a valid login does not mean the account may access everything, and scraping behind a login without permission can breach a site’s terms or the law. Use official APIs and scoped credentials.

Proxy

A proxy is an intermediary that forwards a request, so the target sees the proxy’s address rather than the client’s. Proxies are used for architecture, caching, and network policy; using them to evade a site’s access controls or bans is an ethical and often contractual problem. See proxy types.

IP rotation

IP rotation is the practice of sending requests through a changing set of IP addresses. It is a technique, not a permission slip: rotating addresses to bypass rate limits or bans can violate terms and may be unlawful. Prefer official APIs and negotiated access.

Browser fingerprinting

Browser fingerprinting is the inference of a client’s identity from properties such as canvas rendering, fonts, timezone, and hardware concurrency. Sites use it for fraud prevention; crawlers should not try to defeat it. A headless browser is detectable, and pretending otherwise is a losing strategy.

CAPTCHA

A CAPTCHA is a challenge intended to distinguish humans from automated clients. This site does not teach how to defeat one on behalf of others. If a site presents a CAPTCHA, the respectful paths are an official API, a permission agreement, or stopping. See the responsible crawling checklist.

Deduplication

Deduplication is the removal of repeated URLs or repeated content from a crawl or dataset. URL normalisation handles the same page under different URLs; content fingerprints handle the same content under different URLs. Both are needed. See deduplication.

Incremental crawl

An incremental crawl fetches only what has changed since the previous run, using timestamps, ETags, sitemaps, or content hashes. It is the difference between a crawler that scales and one that re-downloads the same corpus every night. See incremental crawling and change detection.

Data quality

Data quality is the degree to which extracted data is complete, valid, consistent, and fit for its intended use. For scraped data, the practical checks are schema validation, required fields, type and range checks, duplicate detection, and a sample reviewed by a human.

Storage format

A storage format is how extracted records are written, such as CSV, JSON Lines, Parquet, SQLite, or a relational database. JSON Lines and Parquet are common for crawling because they append well and preserve nested fields; a database is better when you need queries, constraints, and concurrent writers.

ETL

ETL (extract, transform, load) is the pipeline pattern of pulling data from sources, cleaning and reshaping it, and loading it into a destination. A crawler is usually the extract step; the transform step is where validation, normalisation, and deduplication belong.