Headless browsers for scraping: when to use one, what it costs, and how to keep it stable

A headless browser is a full browser engine driven by code without a visible window. It executes JavaScript, builds the DOM, runs layout, and exposes the result through an automation protocol. It is the right tool when the data only exists after script execution; it is the wrong tool for a page that already returns its data in HTML.

When a browser is necessary

Use a browser when at least one of these is true:

• The content is assembled in the client from many requests or generated at runtime.
• You need to click, type, scroll, hover, or wait for a user interaction.
• The site requires a real browser to render the permitted content.
• You need a screenshot, a PDF, or a download that only the browser can produce.

Do not use a browser when:

• The HTML contains the data (use an HTTP client and a parser).
• A single JSON/XHR request returns the data (call the endpoint).
• The site publishes an official API (use the API).
• You are trying to bypass a login, paywall, CAPTCHA, or ban (a browser does not grant permission).

The static vs JavaScript lesson shows the one-minute test.

What it costs

A headless Chromium page is not a lightweight HTTP request. A realistic budget:

On a 2-vCPU / 12 GB VM, a browser pool of two pages is a sensible starting point for a background job. Fifty parallel pages will exhaust the machine and make every page slower.

Browser, context, page

The lifecycle is hierarchical:

• Browser: one process (or a small process tree). Expensive to start, reuse it.
• Context: an isolated cookie/storage jar. Cheap. Use one per job or per identity.
• Page: a tab. The unit you navigate. Close it when done.

import { chromium } from 'playwright'

const browser = await chromium.launch()
try {
  const context = await browser.newContext({
    userAgent: 'CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)',
    viewport: { width: 1280, height: 800 },
  })
  const page = await context.newPage()
  page.setDefaultTimeout(15_000)
  page.setDefaultNavigationTimeout(30_000)
  await page.goto('http://127.0.0.1:8765/csr', { waitUntil: 'domcontentloaded' })
  await page.locator('h1').waitFor({ state: 'visible' })
  console.log(await page.locator('h1').innerText())
  await context.close()
} finally {
  await browser.close()
}

Output against the local client-rendered fixture:

Mechanical keyboard

The finally matters. A thrown error without it leaves Chromium running, and the next run starts with less memory.

Stability rules

1. Wait for an element, not for time. waitForSelector or a locator wait. Fixed sleeps are the main source of flakiness.
2. Set timeouts. A navigation timeout and a default action timeout prevent a hung page from blocking the worker.
3. Block unnecessary resources. Images, fonts, and media are often irrelevant to the data:

await context.route('**/*', route => {
  const type = route.request().resourceType()
  if (['image', 'font', 'media'].includes(type)) return route.abort()
  return route.continue()
})

1. Recycle after N pages. Long-lived pages leak memory. Close the page/context after a batch; restart the browser after a larger batch.
2. Screenshot on failure. A screenshot tells you whether the page was blocked, cookie-walled, or redesigned.
3. Cap concurrency by memory, not by hope. Measure the resident set size of the browser process under load.

Detection is not the point

Headless browsers are detectable. Automation flags, missing plugins, unusual navigator properties, canvas differences, and traffic patterns all leave traces. Trying to defeat every signal is an arms race, and on many sites it violates the terms.

The defensible approach is to identify honestly, respect robots.txt and the terms, use an official API or ask for permission, and stop when the site says no. A browser is a rendering tool, not an authorisation.

Browser vs HTTP: a decision table

Sources

• Playwright documentation
• Puppeteer documentation
• Chrome DevTools Protocol