Skip to content

Learn

Static vs JavaScript rendering: how to tell what a crawler actually receives

A server-rendered page and a client-rendered app look the same in a browser and completely different to a crawler. Learn how to tell, when to use a headless browser, and what it costs.

Last updated 4 min readBy CrawlerBros Editorial

Tested with Python 3.14.2, requests 2.32.5, Playwright 1.64.1, Chromium 156 on 2 November 2026. Library versions and dates are part of the evidence, not decoration.

Server-side rendering sends HTML that already contains the page content; client-side rendering sends a shell and builds the content in the browser with JavaScript. A crawler using an HTTP client sees the first; a crawler using a headless browser sees the second after the script runs. The difference decides whether you need a browser at all.

The four common shapes

Shape What the server sends What the crawler sees
Static HTML Full content Full content
Server-rendered app (SSR) Full content, then hydrates Full content
Client-rendered app (CSR) Empty root plus scripts Empty root unless it runs JavaScript
Hybrid / streaming Shell plus some content, then more Partial content; may need a browser for the rest

A page can be hybrid: the product name and price are server-rendered, the reviews load from an API after paint. You may be able to scrape the important fields without a browser and fetch the reviews from their endpoint.

How to tell in under a minute

  1. Fetch the URL with curl or requests.
  2. Search the response for a string you can see in the browser.
  3. If it is absent, open DevTools → Network → Fetch/XHR and reload.
  4. Look for a request whose response contains the missing data.
  5. If the data is in a JSON response, fetch that endpoint. If it is assembled from many requests or generated in the client, consider a browser.
import requests

url = "http://127.0.0.1:8765/csr"
response = requests.get(url, timeout=10, headers={"User-Agent": "CrawlerBros-Example/1.0"})
print(response.status_code)
print("Mechanical keyboard" in response.text)

Tested on 2 November 2026 against a local client-rendered fixture:

200
False

The response is a 200 and the page looks right in a browser, but the product name is not in the HTML. A parser would return zero records and the status code would look healthy.

The same page through a browser

from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://127.0.0.1:8765/csr", wait_until="networkidle")
    print(page.locator("h1").inner_text())
    browser.close()

Tested on 2 November 2026 with Playwright 1.64.1 and Chromium 156:

Mechanical keyboard

wait_until="networkidle" waits until there have been no network connections for 500 ms. It is convenient and fragile: a page with a polling request never becomes idle, and a page that loads data after idle will be read too early. A better wait is for the element you need:

page.goto("http://127.0.0.1:8765/csr")
page.locator("h1").wait_for(state="visible", timeout=10_000)
print(page.locator("h1").inner_text())

Wait for the condition, not for a fixed number of seconds. Fixed sleeps are the most common cause of flaky browser scrapers.

What a browser costs

Resource HTTP client Headless Chromium
Memory per worker A few MB 150–400 MB, more for heavy pages
Startup Milliseconds Hundreds of milliseconds to seconds
CPU Low High; JavaScript, layout, paint
Concurrency Hundreds of requests A handful of pages per core
Failure modes Timeouts, status codes Crashes, memory leaks, selector timing
Debugging Response body Screenshots, traces, console logs

A browser is the right tool when the page genuinely needs one. It is the wrong tool for a JSON endpoint, a static article, or a list of links. The headless browsers guide covers the operational side: pooling, timeouts, memory, and cleanup.

What a browser still does not give you

  • Access. A browser cannot make a login, paywall, or CAPTCHA permissible. It only makes the request look more like a person’s. If the content is behind an access control you do not have permission to pass, stop.
  • Immunity from detection. Headless browsers are detectable through many signals. Trying to defeat detection is an arms race and, on many sites, a terms-of-service problem.
  • Stability. A selector that depends on rendered layout is more brittle than a JSON field. Prefer the API when one exists.

A decision procedure

  1. Is there an official API? Use it.
  2. Is the data in the HTML? Use an HTTP client and a parser.
  3. Is the data in a JSON/XHR request the page makes? Fetch that request directly, with the same headers and parameters a browser would send.
  4. Is the data built entirely in the client, with no single data request? Use a headless browser, cache the rendered HTML, and keep the browser pool small.
  5. Does the site’s terms or robots.txt prohibit the access? Stop, or ask for permission.

Most “we need Playwright” projects stop at step 3 once someone opens the network panel.

Rendering and SEO

The same distinction matters for your own site. If you publish content that only exists after JavaScript runs, a crawler that does not execute scripts sees an empty page. This site prerenders every route to static HTML for that reason: the content is in the response, and JavaScript is an enhancement. The prerender notes explain the policy.

Key takeaways

  • A 200 response can still be an empty shell.
  • Search the response for the content before reaching for a browser.
  • Prefer the data endpoint over rendering the page that calls it.
  • Wait for the element, not for a fixed sleep.
  • A browser does not grant permission; it only executes JavaScript.
  • Budget for memory and CPU when you choose a browser.

Sources checked