Skip to content

Guides

Python web scraping with requests and BeautifulSoup

Build a small crawler in Python with requests and BeautifulSoup: frontier, deduplication, robots.txt, per-host delay, pagination, JSONL storage, and the failure checks that matter.

Last updated 4 min readBy CrawlerBros Editorial

Tested with Python 3.14.2, requests 2.32.5, BeautifulSoup 4.14.3 on 2 November 2026. Library versions and dates are part of the evidence, not decoration.

requests fetches a URL and returns the response; BeautifulSoup parses the HTML into a tree you can query. Together they are the smallest serious Python scraping stack: no browser, no framework, no hidden magic. This guide builds a complete crawler with a frontier, deduplication, robots.txt, a per-host delay, pagination, and JSONL output.

The code is tested against a local fixture, so every URL and status is deterministic. Replace the seed and selectors with your own only after the fixture version works.

Setup

Use a virtual environment so the versions are yours, not the system's.

python -m venv .venv
# Linux/macOS
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install requests beautifulsoup4

Tested with Python 3.14.2, requests 2.32.5, and BeautifulSoup 4.14.3 on 2 November 2026.

Fetch one page correctly

import requests

URL = "http://127.0.0.1:8765/"
HEADERS = {
    "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    "Accept": "text/html,application/xhtml+xml",
}

response = requests.get(URL, headers=HEADERS, timeout=10)
response.raise_for_status()
print(response.status_code, response.headers.get("content-type"))
print(response.text[:60])

Output against the fixture:

200 text/html; charset=utf-8
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Fixt

Three habits are baked in: a timeout, a descriptive user agent, and a status check before parsing. Without the timeout, one stalled socket can hang the run. Without the status check, a 404 error page parses as a page with no data, which looks like a selector bug.

Parse the page into links

from bs4 import BeautifulSoup
from urllib.parse import urljoin

soup = BeautifulSoup(response.text, "html.parser")
for link in soup.select("a[href]"):
    print(urljoin(URL, link["href"]))

Output:

http://127.0.0.1:8765/about
http://127.0.0.1:8765/guides
http://127.0.0.1:8765/items?page=1

urljoin resolves relative links. A site that uses /about and about in different places will otherwise produce two different strings for the same page.

Add a frontier and deduplication

A frontier is a queue of URLs to fetch. seen prevents the same URL from being queued twice. Without it, a page that links to itself will loop until the process dies.

from collections import deque

queue = deque([URL])
seen = {URL}

while queue:
    current = queue.popleft()
    response = requests.get(current, headers=HEADERS, timeout=10)
    if response.status_code != 200:
        continue
    soup = BeautifulSoup(response.text, "html.parser")
    for link in soup.select("a[href]"):
        absolute = urljoin(current, link["href"])
        if absolute not in seen:
            seen.add(absolute)
            queue.append(absolute)

This is a breadth-first crawl. It has no scope limit, no robots check, no delay, and no stop condition. Add those before pointing it at a real site.

Read robots.txt and apply it

urllib.robotparser is in the standard library. It reads /robots.txt and answers can_fetch(user_agent, url). It is a reasonable first implementation; for production, check that it supports the wildcard patterns your target uses, or use a library such as protego that implements RFC 9309 more completely.

from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser

parsed = urlparse(URL)
root = f"{parsed.scheme}://{parsed.netloc}"
robots = RobotFileParser()
robots.set_url(f"{root}/robots.txt")
robots.read()

print(robots.can_fetch(HEADERS["User-Agent"], f"{root}/private"))
print(robots.can_fetch(HEADERS["User-Agent"], f"{root}/guides"))

Output against the fixture:

False
True

Treat robots.txt as a floor, not a ceiling. The terms of service, copyright, privacy law, and technical access controls still apply. The robots.txt guide covers the matching rules in detail.

Add a delay and a stop condition

A delay belongs between requests to the same host, not after every request in a multi-host crawl. This example crawls one host, so a simple sleep is enough. It also stops after 10 pages so a mistake cannot run forever.

import time

MAX_PAGES = 10
DELAY_SECONDS = 0.5
records = []

while queue and len(records) < MAX_PAGES:
    current = queue.popleft()
    if not robots.can_fetch(HEADERS["User-Agent"], current):
        continue
    response = requests.get(current, headers=HEADERS, timeout=10)
    if response.status_code != 200 or "text/html" not in response.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(response.text, "html.parser")
    records.append({
        "url": current,
        "title": soup.title.get_text(strip=True) if soup.title else "",
        "status": response.status_code,
    })
    for link in soup.select("a[href]"):
        absolute = urljoin(current, link["href"])
        if urlparse(absolute).netloc == parsed.netloc and absolute not in seen:
            seen.add(absolute)
            queue.append(absolute)
    time.sleep(DELAY_SECONDS)

The netloc check keeps the crawl on one host. Without it, the first external link takes the crawler off-site, and the delay no longer matches the host you intended to be polite to.

Store the records

JSON Lines is the simplest durable format for a crawl. Each line is a complete record, so a crash loses at most the last line, and you can append without rewriting the file.

import json

with open("pages.jsonl", "w", encoding="utf-8") as handle:
    for record in records:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")

print(len(records), "records")

Appending is usually better than opening with "w" on every run. Use a run ID and a timestamp so you can tell runs apart, and write to a temporary file before renaming it if you need atomic replacement.

The complete script

import json
import time
from collections import deque
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

START = "http://127.0.0.1:8765/"
USER_AGENT = "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"
MAX_PAGES = 10
DELAY_SECONDS = 0.5

session = requests.Session()
session.headers["User-Agent"] = USER_AGENT

parsed = urlparse(START)
root = f"{parsed.scheme}://{parsed.netloc}"
robots = RobotFileParser()
robots.set_url(f"{root}/robots.txt")
robots.read()

queue = deque([START])
seen = {START}
records = []

while queue and len(records) < MAX_PAGES:
    url = queue.popleft()
    if not robots.can_fetch(USER_AGENT, url):
        continue

    response = session.get(url, timeout=10)
    content_type = response.headers.get("content-type", "")
    if response.status_code != 200 or "text/html" not in content_type:
        continue

    soup = BeautifulSoup(response.text, "html.parser")
    records.append({
        "url": url,
        "title": soup.title.get_text(strip=True) if soup.title else "",
        "status": response.status_code,
    })

    for link in soup.select("a[href]"):
        absolute = urljoin(url, link["href"])
        if urlparse(absolute).netloc == parsed.netloc and absolute not in seen:
            seen.add(absolute)
            queue.append(absolute)

    time.sleep(DELAY_SECONDS)

with open("pages.jsonl", "w", encoding="utf-8") as handle:
    for record in records:
        handle.write(json.dumps(record, ensure_ascii=False) + "\n")

print(json.dumps(records, indent=2))

Tested on 2 November 2026 against the fixture. Output:

[
  {
    "url": "http://127.0.0.1:8765/",
    "title": "Fixture home",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/about",
    "title": "About",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/guides",
    "title": "Guides",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/items?page=1",
    "title": "Items page 1",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/guides/http",
    "title": "HTTP for crawlers",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/guides/selectors",
    "title": "Selectors",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/items?page=2",
    "title": "Items page 2",
    "status": 200
  },
  {
    "url": "http://127.0.0.1:8765/items?page=3",
    "title": "Items page 3",
    "status": 200
  }
]

The exact order depends on the link order in the HTML and the queue. That is fine: assert on the set of URLs, not on the order, in your tests.

Extracting fields, not just pages

Once the crawl works, add a parser that turns each page into records. Keep the parser separate from the fetch loop so you can test it on saved HTML.

def parse_items(html: str) -> list[dict[str, str]]:
    soup = BeautifulSoup(html, "html.parser")
    items = []
    for card in soup.select("ul.results li.item"):
        items.append({
            "sku": card["data-sku"],
            "name": card.select_one("h3").get_text(strip=True),
            "price": card.select_one(".price").get_text(strip=True),
            "stock": card.select_one(".stock").get_text(strip=True),
        })
    return items

Validate the result: if a page should have 20 items and the parser returns 0, fail the run instead of writing an empty file. A selector that silently returns nothing is worse than a crash.

Production notes

  • Retries. Retry connection errors, timeouts, 429, and most 5xx with backoff. Do not retry 400, 401, 403, or 404 unchanged. See retries and idempotency.
  • Concurrency. A single-threaded loop is enough for a few thousand pages. For more, use a small thread pool with a per-host limiter, or move to Scrapy, which has a scheduler and throttling built in.
  • Pagination. Follow the next link, not a guessed ?page=N. Stop when the next link is missing or the page repeats. See pagination and infinite scroll.
  • Conditional requests. Store ETag and Last-Modified and send them back next run. A 304 saves the body. See incremental crawling.
  • Storage. JSONL for raw pages, a database for queryable records. Keep the raw HTML for a short window so a parser bug can be fixed without re-crawling.
  • Monitoring. Log status counts, parse counts, and duration. Alert on a sudden drop in records per page; that is usually a layout change, not a network problem.

Sources

Sources checked