Scrapy vs Crawlee: which crawling framework fits your stack?

Scrapy is a Python crawling framework built around a Twisted event loop, a scheduler, duplicate filtering, item pipelines, and feed exports. Crawlee is a Node.js crawling library built around a request queue, autoscaling, session and proxy management, and interchangeable HTTP/browser crawlers. Both are mature; the choice is usually about language and team fit, not raw capability.

We did not run a head-to-head throughput benchmark. A fair benchmark would need the same corpus, the same concurrency, the same parser work, and the same network conditions; a number from a different machine and fixture would mislead. The table describes documented architecture.

At a glance

Criterion	Scrapy	Crawlee
Language	Python	TypeScript/JavaScript (Node.js)
Core model	Scheduler + downloader + spiders + pipelines	Request queue + crawlers + datasets
Default parsing	Parsel (CSS/XPath)	Cheerio, Playwright, Puppeteer
Browser support	Via `scrapy-playwright` or Splash	First-class Playwright/Puppeteer crawlers
Throttling	`DOWNLOAD_DELAY`, AutoThrottle, concurrency settings	`maxConcurrency`, `maxRequestsPerMinute`, autoscaling
Duplicate filtering	Built in	Built in (request queue)
Storage	Feeds (JSONL/CSV/XML), pipelines to DB	Datasets, key-value store, request queue
Retry policy	Configurable middleware	Configurable per crawler
Ecosystem	Large, long-established Python ecosystem	Modern Node.js ecosystem, Apify-backed
Learning curve	Steeper: settings, middleware, Twisted	Gentler for a small Node script; browser crawlers add complexity

Architecture

Scrapy’s unit is the spider: a class with start_urls and parse methods that yield items or requests. The engine schedules requests, the downloader fetches them, the spider parses, and pipelines process items. Settings are global with per-spider overrides. Middleware lets you change requests, responses, retries, and cookies.

Crawlee’s unit is the crawler: CheerioCrawler for HTTP + Cheerio, PlaywrightCrawler/PuppeteerCrawler for a browser. A requestHandler receives the request and parsed page, calls enqueueLinks, and pushes records to a dataset. The request queue provides de-duplication and retries; a session pool and proxy configuration are available.

Both models are sound. Scrapy feels like a framework you configure; Crawlee feels like a library you compose. A team already writing Python will be productive in Scrapy faster. A team already writing Node.js will be productive in Crawlee faster.

Politeness and throttling

Scrapy’s AUTOTHROTTLE_ENABLED adjusts delay from latency and concurrency. DOWNLOAD_DELAY sets a floor. CONCURRENT_REQUESTS_PER_DOMAIN caps per-domain concurrency. ROBOTSTXT_OBEY enables robots.txt handling.

Crawlee’s maxConcurrency caps in-flight requests, maxRequestsPerMinute sets a global rate, and the autoscaled pool adjusts concurrency based on system load. Per-domain limits need configuration; read the current API rather than assuming a default.

Neither framework knows your target’s tolerance. Start conservative, measure, and raise concurrency only when latency and error rates stay healthy.

Browser work

Crawlee treats browser crawlers as first-class: you can swap CheerioCrawler for PlaywrightCrawler and keep the queue, storage, and retry model. Scrapy can drive a browser through scrapy-playwright, but the browser integration is an add-on and the performance profile changes substantially.

If most of your targets are client-rendered, Crawlee’s browser story is simpler. If most are server-rendered and you want a Python pipeline, Scrapy is the more natural fit.

Storage and pipelines

Scrapy feeds write items to JSONL, CSV, or XML, and pipelines can write to a database, validate fields, or enrich records. Crawlee datasets write JSON lines by default and expose a key-value store; you can push to an external database in the request handler or a separate step.

Both approaches encourage keeping storage out of the parsing code. Do that. A spider or crawler that writes directly to production tables is hard to test and hard to replay.

When to choose Scrapy

Your team writes Python.
You want a mature ecosystem of middlewares and pipelines.
The targets are mostly server-rendered.
You need fine-grained control over scheduling and item processing.
You want a single framework that has been deployed in production for years.

When to choose Crawlee

Your team writes TypeScript/JavaScript.
You need to switch between HTTP and browser crawling with the same queue and storage.
You want a local dataset and request queue with minimal setup.
You are already deploying Node.js services.

When neither is the right tool

One page or a handful of requests. Use requests/httpx or fetch.
A documented API. Use the API client.
A single browser interaction. A small Playwright script is clearer than a framework.
A crawl that must run inside an existing event loop. Scrapy owns its reactor; Crawlee runs on Node’s event loop but still expects to own the crawl lifecycle.