Playwright web scraping in JavaScript: waits, pagination, and browser hygiene

Playwright drives Chromium, Firefox, and WebKit through a single API and is designed around auto-waiting locators. Use it when a page genuinely needs JavaScript to produce the data, or when you need browser behaviour such as clicks, downloads, or a rendered screenshot. Do not use it for a page that already returns the data in its HTML.

This guide uses a local fixture and writes the real output. The important parts are the waits, the browser lifecycle, and the decision to use a browser at all.

Install

mkdir pw-scrape && cd pw-scrape
npm init -y
npm pkg set type=module
npm install playwright
npx playwright install chromium

Tested with Node.js 24.13.0, Playwright 1.64.1, and Chromium 156 on 2 November 2026. npx playwright install chromium downloads the browser binary; it is a separate step from installing the npm package.

Launch, navigate, extract

import { chromium } from 'playwright'

const browser = await chromium.launch()
const context = await browser.newContext({
  userAgent: 'CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)',
})
const page = await context.newPage()

await page.goto('http://127.0.0.1:8765/items?page=1', { waitUntil: 'domcontentloaded' })
await page.locator('ul.results li.item').first().waitFor({ state: 'visible' })

const first = page.locator('ul.results li.item').first()
console.log(await first.locator('h3').innerText())
console.log(await first.locator('.price').innerText())

await context.close()
await browser.close()

Output against the fixture:

Mechanical keyboard
$89.00

waitUntil: 'domcontentloaded' returns when the HTML is parsed, not when every image or analytics request has finished. The locator’s waitFor then waits for the element you actually need. That is more reliable than networkidle, which never settles on a page with polling or long-lived connections.

Pagination

import { chromium } from 'playwright'

const browser = await chromium.launch()
const context = await browser.newContext({
  userAgent: 'CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)',
})
const page = await context.newPage()
await page.goto('http://127.0.0.1:8765/items?page=1', { waitUntil: 'domcontentloaded' })

const records = []
while (true) {
  await page.locator('ul.results li.item').first().waitFor({ state: 'visible' })
  const cards = page.locator('ul.results li.item')
  const count = await cards.count()
  for (let index = 0; index < count; index += 1) {
    const card = cards.nth(index)
    records.push({
      sku: await card.getAttribute('data-sku'),
      name: await card.locator('h3').innerText(),
      price: await card.locator('.price').innerText(),
    })
  }
  const next = page.locator('a[rel="next"]')
  if ((await next.count()) === 0) break
  await next.click()
  await page.waitForLoadState('domcontentloaded')
}

console.log(JSON.stringify(records, null, 2))
await context.close()
await browser.close()

Output against the fixture:

[
  {
    "sku": "A-1",
    "name": "Mechanical keyboard",
    "price": "$89.00"
  },
  {
    "sku": "B-2",
    "name": "USB-C hub",
    "price": "$39.50"
  },
  {
    "sku": "C-3",
    "name": "Laptop stand",
    "price": "$54.00"
  },
  {
    "sku": "D-4",
    "name": "Noise-cancelling headphones",
    "price": "$199.00"
  },
  {
    "sku": "E-5",
    "name": "Webcam",
    "price": "$69.00"
  }
]

The loop stops when the rel="next" link disappears. It does not guess ?page=2, and it does not keep clicking a disabled button. A repeated page is the other stop condition: hash the extracted records and break if the hash has not changed.

Wait for the right thing

Wait for a fact, not for time. If the data arrives in a JSON response, wait for that response:

const responsePromise = page.waitForResponse(
  response => response.url().includes('/api/items') && response.status() === 200,
)
await page.goto('http://127.0.0.1:8765/', { waitUntil: 'domcontentloaded' })
const response = await responsePromise
const payload = await response.json()
console.log(payload.items.length)

If the page makes no such request, the data is built in the client and a locator wait is the right tool.

Read the network before rendering

A browser is expensive. Before you write a Playwright crawler, open DevTools → Network and look for the request that returns the data. If it is a JSON endpoint, call it with an HTTP client. The XHR and GraphQL discovery guide shows how to find it and what to copy.

Use a browser when:

• the data is assembled from many requests or generated in the client;
• you need to click, type, scroll, or download;
• the site requires a real browser to render the content you are permitted to access.

Do not use a browser to defeat a login, paywall, CAPTCHA, or ban. A browser changes how the request is made, not whether it is allowed.

Browser hygiene

• Close what you open. context.close() and browser.close() in a finally block. A crashed run that leaves Chromium processes behind will exhaust memory.
• Reuse the browser, not the page. Launch one browser per worker, create a context per job, and a page per URL. Contexts isolate cookies and storage.
• Cap concurrency. One Chromium page can use hundreds of MB. Start with one page per CPU core and measure; do not run 50 pages on a 2-vCPU box.
• Set timeouts. page.setDefaultTimeout(15000) and page.setDefaultNavigationTimeout(30000) prevent a single page from hanging the run.
• Block what you do not need. Route and abort images, fonts, and media if they are not part of the data. It saves bandwidth and time:

await context.route('**/*', route => {
  const type = route.request().resourceType()
  if (type === 'image' || type === 'font' || type === 'media') return route.abort()
  return route.continue()
})

• Take a screenshot on failure. It is the fastest way to see whether the page was a cookie banner, a block page, or a layout change.

try {
  await page.goto(url, { waitUntil: 'domcontentloaded' })
  await page.locator('ul.results li.item').first().waitFor({ state: 'visible' })
} catch (error) {
  await page.screenshot({ path: `failure-${Date.now()}.png`, fullPage: true })
  throw error
}

Downloads and screenshots

Playwright can capture a PDF or a screenshot, but a screenshot is not data. If you need a visual record, set an explicit viewport and use fullPage: true. If you need a file, use the download event and save it to a path you control; do not trust a filename supplied by the page.

Detection and honesty

Headless browsers are detectable through many signals: missing plugins, unusual navigator values, automation flags, canvas differences, and network behaviour. Trying to defeat detection is an arms race, and on many sites it is a terms-of-service problem.

The honest approach:

• use a descriptive user agent;
• do not pretend to be a logged-in user;
• respect robots.txt and the terms;
• use an official API or ask for permission when the data is behind an access control;
• stop when the site says no.

A browser is a tool for rendering, not a licence to access.

Sources

• Playwright documentation
• Locators
• Network