Crawlee tutorial: a JavaScript crawler with a queue, retries, and storage

Crawlee is a Node.js crawling library with a request queue, automatic retries, concurrency control, session management, and dataset storage. CheerioCrawler parses HTML without a browser; PlaywrightCrawler and PuppeteerCrawler run a real browser when the page needs one.

This guide uses the Cheerio crawler against a local fixture. It is the right default for server-rendered pages and the cheapest way to learn Crawlee’s queue model.

Install

mkdir crawlee-scrape && cd crawlee-scrape
npm init -y
npm pkg set type=module
npm install crawlee

Tested with Node.js 24.13.0 and Crawlee 3.18.2 on 2 November 2026.

The smallest crawler

import { CheerioCrawler } from 'crawlee'

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 10,
  requestHandler: async ({ request, $, enqueueLinks }) => {
    console.log(request.url, $('title').text().trim())
    await enqueueLinks({ strategy: 'same-domain' })
  },
})

await crawler.run(['http://127.0.0.1:8765/'])

Output against the fixture:

http://127.0.0.1:8765/ Fixture home
http://127.0.0.1:8765/guides Guides
http://127.0.0.1:8765/about About
http://127.0.0.1:8765/guides/http HTTP for crawlers
http://127.0.0.1:8765/items?page=1 Items page 1
http://127.0.0.1:8765/guides/selectors Selectors
http://127.0.0.1:8765/items?page=2 Items page 2
http://127.0.0.1:8765/items?page=3 Items page 3

enqueueLinks({ strategy: 'same-domain' }) discovers links and adds them to the queue. Crawlee’s queue de-duplicates URLs, retries failures, and stores state under ./storage by default. The order is not guaranteed; assert on sets.

Extract records into a dataset

import { CheerioCrawler } from 'crawlee'

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 10,
  requestHandler: async ({ $, enqueueLinks, pushData }) => {
    for (const card of $('ul.results li.item').toArray()) {
      const element = $(card)
      await pushData({
        sku: element.attr('data-sku'),
        name: element.find('h3').text().trim(),
        price: element.find('.price').text().trim(),
        stock: element.find('.stock').text().trim(),
      })
    }
    await enqueueLinks({ strategy: 'same-domain' })
  },
})

await crawler.run(['http://127.0.0.1:8765/items?page=1'])
const dataset = await crawler.getData()
console.log(JSON.stringify(dataset.items, null, 2))

The dataset is stored as JSON lines under storage/datasets/default/. pushData is async; await it so a storage failure is not silently lost.

Control concurrency and politeness

const crawler = new CheerioCrawler({
  minConcurrency: 1,
  maxConcurrency: 4,
  maxRequestsPerMinute: 60,
  requestHandler: async ({ request, $, enqueueLinks }) => {
    console.log(request.url, $('title').text().trim())
    await enqueueLinks({ strategy: 'same-domain' })
  },
})

maxConcurrency caps in-flight requests.
maxRequestsPerMinute is a simple global rate limit.
minConcurrency lets Crawlee scale down when the target is slow.

For per-domain limits, use maxRequestsPerMinute with a single-domain crawl, or a custom AutoscaledPool/SessionPool configuration. Crawlee’s defaults are reasonable; do not raise concurrency until you have measured the target’s response times.

Retries and error handling

Crawlee retries failed requests by default. Tune it explicitly:

const crawler = new CheerioCrawler({
  maxRequestRetries: 2,
  retryOnBlocked: true,
  requestHandler: async ({ request, $, enqueueLinks, log }) => {
    if ($('title').text().includes('Access denied')) {
      log.warning(`blocked: ${request.url}`)
      return
    }
    await enqueueLinks({ strategy: 'same-domain' })
  },
  failedRequestHandler: async ({ request, error }) => {
    console.error(`failed: ${request.url}`, error.message)
  },
})

Do not retry a 403 or a 404 and expect a different answer. A 403 usually means the request is not permitted or the site has blocked you; a 404 is a missing page. Retry 429, 5xx, timeouts, and connection errors with backoff. See retries and idempotency.

Browser crawling

PlaywrightCrawler uses the same queue and dataset model with a real browser:

import { PlaywrightCrawler } from 'crawlee'

const crawler = new PlaywrightCrawler({
  maxConcurrency: 1,
  requestHandler: async ({ page, enqueueLinks, pushData }) => {
    await page.waitForSelector('h1')
    await pushData({ url: page.url(), title: await page.locator('h1').innerText() })
    await enqueueLinks({ strategy: 'same-domain' })
  },
})

The browser is the expensive part. Keep maxConcurrency low, reuse the browser, and close contexts. If the data is in a JSON endpoint, use CheerioCrawler or a plain HTTP client instead.

When Crawlee is the right choice

You want a queue, retries, and storage without building them.
You are already in Node.js and want one library for HTTP and browser crawling.
You need session handling and proxy rotation through a supported API.
You want a local dataset you can inspect after a run.

It is not the right choice for a single request, a documented API, or a crawl that must fit inside another framework’s event loop.