Skip to content

Guides

Scrapy tutorial: a production-shaped spider with robots.txt, throttling, and feeds

Build and run a Scrapy spider that obeys robots.txt, throttles itself, follows links, and writes JSON Lines. Includes the settings that matter and the mistakes that cause bans.

Last updated 4 min readBy CrawlerBros Editorial

Tested with Python 3.14.2, Scrapy 2.19.0 on 2 November 2026. Library versions and dates are part of the evidence, not decoration.

Scrapy is a Python crawling framework with a scheduler, duplicate filter, request throttling, item pipelines, and feed exports built in. You write spiders that parse responses and yield items or new requests; Scrapy handles the queue, concurrency, retries, and politeness settings.

This guide uses a local fixture so the output is deterministic. The spider is deliberately small; the value is in the settings and the structure, not in the number of lines.

Install and create a project

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install scrapy
scrapy startproject fixture_scrapy
cd fixture_scrapy

Tested with Python 3.14.2 and Scrapy 2.19.0 on 2 November 2026. Scrapy pulls in Twisted, lxml, parsel, and w3lib; the exact versions are printed at startup, which is useful when you report a bug.

The spider

Create fixture_scrapy/spiders/fixture.py:

import scrapy


class FixtureSpider(scrapy.Spider):
    name = "fixture"
    start_urls = ["http://127.0.0.1:8765/"]

    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 0.5,
        "AUTOTHROTTLE_ENABLED": True,
        "AUTOTHROTTLE_TARGET_CONCURRENCY": 1.0,
        "CLOSESPIDER_PAGECOUNT": 10,
        "FEEDS": {"pages.jsonl": {"format": "jsonlines", "overwrite": True}},
        "LOG_LEVEL": "ERROR",
    }

    def parse(self, response):
        yield {
            "url": response.url,
            "title": response.css("title::text").get(default="").strip(),
        }
        for href in response.css("a[href]::attr(href)").getall():
            yield response.follow(href, callback=self.parse)

Run it from the project root:

scrapy crawl fixture

Output in pages.jsonl against the fixture:

{"url": "http://127.0.0.1:8765/", "title": "Fixture home"}
{"url": "http://127.0.0.1:8765/items?page=1", "title": "Items page 1"}
{"url": "http://127.0.0.1:8765/guides", "title": "Guides"}
{"url": "http://127.0.0.1:8765/about", "title": "About"}
{"url": "http://127.0.0.1:8765/items?page=2", "title": "Items page 2"}
{"url": "http://127.0.0.1:8765/guides/selectors", "title": "Selectors"}
{"url": "http://127.0.0.1:8765/guides/http", "title": "HTTP for crawlers"}
{"url": "http://127.0.0.1:8765/items?page=3", "title": "Items page 3"}

response.follow resolves relative URLs and carries the callback. Scrapy’s duplicate filter keeps the same URL from being scheduled twice within a run. The order is not guaranteed; assert on sets, not order.

The settings that matter

ROBOTSTXT_OBEY = True makes Scrapy fetch and obey /robots.txt. It is the first setting to turn on and the first one people turn off when a crawl returns fewer pages than expected. If robots.txt is blocking you, that is the answer, not an obstacle.

DOWNLOAD_DELAY = 0.5 sets a minimum delay between requests to the same domain. It is a blunt instrument but a good default. Combine it with AutoThrottle rather than using a large fixed delay for every site.

AUTOTHROTTLE_ENABLED = True adjusts the delay from observed latency and concurrency. AUTOTHROTTLE_TARGET_CONCURRENCY is the average number of requests Scrapy tries to keep in flight per domain. Start at 1.0 and raise it only after measuring the target site’s tolerance.

CLOSESPIDER_PAGECOUNT is a hard stop. Use it in development and in any crawl where a pagination bug could otherwise run forever.

FEEDS writes items to a file. jsonlines is append-friendly and preserves nested fields. For a production pipeline, write to a database or object storage instead of a local file.

Items and pipelines

An item is a structured record; a pipeline validates, cleans, or stores it. The smallest useful pipeline rejects records missing a required field:

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class RequiredFieldsPipeline:
    required = ("url", "title")

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        missing = [field for field in self.required if not adapter.get(field)]
        if missing:
            raise DropItem(f"missing {', '.join(missing)}")
        return item

Enable it in settings.py:

ITEM_PIPELINES = {
    "fixture_scrapy.pipelines.RequiredFieldsPipeline": 300,
}

A pipeline that silently drops malformed records hides a layout change. Log the drop reason and count drops per run; a sudden increase is a signal that the site changed.

Pagination

Follow the site’s real next link. Do not construct ?page=N unless the site documents it.

def parse(self, response):
    for card in response.css("ul.results li.item"):
        yield {
            "sku": card.attrib.get("data-sku"),
            "name": card.css("h3::text").get(default="").strip(),
            "price": card.css(".price::text").get(default="").strip(),
            "stock": card.css(".stock::text").get(default="").strip(),
        }
    next_page = response.css('a[rel="next"]::attr(href)').get()
    if next_page:
        yield response.follow(next_page, callback=self.parse)

The rel="next" link is a better contract than a guessed page number. Stop when it disappears, when the page returns the same records, or when a cursor repeats.

Retries and errors

Scrapy retries some failures by default. Tune it deliberately:

RETRY_ENABLED = True
RETRY_TIMES = 2
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]
DOWNLOAD_TIMEOUT = 20

Do not add 403 or 404 to RETRY_HTTP_CODES. A 403 will not become a 200 because you asked again; it usually means the request is not permitted or the site has blocked you. A 404 is a missing page. See retries and idempotency for the policy.

Honour Retry-After on 429 and 503. Scrapy’s retry middleware respects it when RETRY_HTTP_CODES includes the status; check the version’s behaviour before relying on it, and add a custom middleware if you need a hard cap.

Caching during development

HTTPCACHE_ENABLED = True stores responses on disk so you can iterate on a parser without re-fetching the site. It is for development, not for production freshness. Clear the cache when you need to see a change.

HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = "httpcache"

A cache is also a debugging tool: when a selector breaks, replay the cached response before blaming the site.

Project layout that scales

fixture_scrapy/
  scrapy.cfg
  fixture_scrapy/
    __init__.py
    items.py
    middlewares.py
    pipelines.py
    settings.py
    spiders/
      fixture.py

Keep selectors in the spider or a small parser module, not scattered across pipelines. Keep storage in a pipeline. Keep settings in settings.py with comments explaining why each non-default value is there. A settings file with twenty unexplained overrides is a bug report waiting to happen.

When Scrapy is the wrong tool

  • One page, one request. requests is simpler.
  • A JavaScript-only page with no data endpoint. Scrapy can drive a browser through scrapy-playwright, but the browser becomes the expensive part; a dedicated Playwright script may be clearer.
  • A documented API. Use the API client, not a spider.
  • A crawl that must run inside an existing async application. Scrapy owns the event loop; embedding it can be awkward.

Sources

Sources checked