What is web crawling? The crawl loop and its limits

Web crawling is the automated process of discovering and fetching web pages by following links and making HTTP requests. Web scraping is the extraction of structured data from the pages a crawler retrieves. A crawler decides what to fetch, in what order, and how politely; a scraper turns a fetched page into fields.

The distinction matters because most failed scraping projects are not parsing failures. They are crawling failures: the wrong URLs, the wrong order, no deduplication, no rate limit, or a page that only exists after JavaScript runs.

Crawling and scraping are different jobs

A single Python script can do all three. A production system usually separates them, because each has a different failure mode. If the crawler fetches the wrong page, a perfect parser cannot save you. If the parser is wrong, a perfect crawler just collects the wrong data faster.

The crawl loop

A crawler repeats five steps:

1. Seed the frontier with known URLs: a homepage, a sitemap, an API endpoint, or a list you already have.
2. Pop a URL according to priority and politeness rules.
3. Fetch it over HTTP, following redirects only as your policy allows, and recording the status.
4. Parse the response, extract the data you need, and discover new URLs.
5. Store the record, the new URLs, and the crawl metadata (status, timestamp, hash, ETag).

Then it repeats until the frontier is empty, a budget is exhausted, or a stop condition is met.

The frontier is the heart of the crawler. It needs three properties:

• Deduplication so the same URL is not fetched repeatedly.
• Priority so important pages are fetched before deep, low-value pages.
• Per-host politeness so one site is not overwhelmed while another waits.

A list of URLs with a for loop is a crawler only in the trivial case. It has no frontier, no deduplication, and no per-host limit.

A minimal fetch, with the parts that matter

The smallest useful example fetches one page, checks the status, parses the HTML, and prints the links. It is not a crawler yet, but it is the fetch-and-parse step every crawler needs.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "http://127.0.0.1:8765/"
HEADERS = {
    "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    "Accept": "text/html,application/xhtml+xml",
}

response = requests.get(BASE, headers=HEADERS, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

for link in soup.select("a[href]"):
    print(urljoin(BASE, link["href"]))

Tested against a local fixture on 2 November 2026. Output:

http://127.0.0.1:8765/about
http://127.0.0.1:8765/guides
http://127.0.0.1:8765/guides/http

Three details are doing real work:

• timeout=10 prevents a stalled connection from hanging the process forever.
• raise_for_status() turns a 404 or 500 into an exception instead of parsing an error page as if it were content.
• urljoin resolves relative links against the base URL; without it, /about is not a fetchable URL.

The User-Agent is descriptive and includes a contact URL. That is the norm for a responsible crawler. Pretending to be Chrome is not a politeness strategy.

What a crawler must respect

Before fetching a path, a crawler should read the site’s robots.txt and apply the matching rules. The protocol is RFC 9309. It is not the only rule: terms of service, copyright, database rights, privacy law, and technical access controls all matter. A path that is allowed by robots.txt can still be off-limits for other reasons.

The practical checklist:

• robots.txt: parse the group for your user agent, fall back to *, apply longest-match with Allow winning ties.
• Rate: cap requests per host, not just per process. Honour Retry-After on 429 and 503.
• Terms: read the site’s terms and the API documentation. If the terms prohibit automated access, stop.
• Personal data: do not collect personal data at scale without a lawful basis and a retention plan.
• Authentication: do not bypass a login or a paywall. Use an official API with scoped credentials.
• Identification: use a real user agent and a contact URL. Do not rotate identities to evade a block.

The responsible crawling checklist turns this into a pre-flight list.

When an API is better than crawling

If the site publishes an API, use it. An API is a contract: it documents fields, rate limits, authentication, and terms. It is usually more stable than HTML, and it is the path the site expects you to take.

Use the API when:

• the data is available through an official endpoint;
• you need a stable schema rather than a layout;
• the site’s terms point developers to the API;
• you need higher volume than a polite HTML crawl can justify.

Crawl the HTML when there is no API, the data is public, the terms allow it, and the volume is modest. Do not build a browser automation pipeline for data that a documented endpoint already returns.

When JavaScript changes the answer

A plain HTTP client receives the HTML the server sends. If the page builds its content with JavaScript, the client sees the shell: a <div id="root"> and a script tag. A browser sees the rendered DOM instead.

Before reaching for a headless browser, open the network panel and look for the request that returns the data. Many “JavaScript-only” pages are a JSON endpoint with a thin UI. Fetching that endpoint is faster, cheaper, and more stable than rendering the page. The static vs JavaScript rendering lesson covers how to tell the difference.

The smallest complete crawler

A complete crawler adds four things to the example above:

1. A queue of URLs and a set of seen URLs.
2. A per-host rate limiter.
3. A robots.txt check before each fetch.
4. A stop condition: a page limit, a depth limit, or a time budget.

The Python requests + BeautifulSoup guide builds that crawler and tests it. The Scrapy guide shows the same ideas in a framework that already provides a frontier, deduplication, throttling, and pipelines.

Key takeaways

• Crawling is discovery and fetching; scraping is extraction. Most failures live in the crawl layer.
• A frontier needs deduplication, priority, and per-host politeness.
• Check the status before parsing; error pages parse surprisingly well.
• Read robots.txt, the terms, and the API documentation before the first request.
• Prefer an official API. Use a headless browser only when the data is not available without one.