Finding XHR and GraphQL endpoints: scrape the data, not the pixels

XHR and fetch are the browser APIs pages use to request data after the HTML loads. GraphQL is a query language and server runtime that often sits behind those requests. If the data arrives in a JSON response, fetch that response directly; it is faster and more stable than rendering the page.

The network-panel workflow

Open DevTools → Network.
Filter to Fetch/XHR.
Reload the page.
Sort by size or look for requests whose response contains the data you can see.
Click the request; read the URL, method, request headers, and payload.
Right-click → Copy as cURL to reproduce it.
Replay it with requests or curl, then strip the headers you do not need one at a time.

The request that returns the data is usually obvious: a JSON response with the same product names, prices, or comments visible on the page.

Replay a JSON endpoint

import requests

response = requests.get(
    "http://127.0.0.1:8765/api/items",
    headers={
        "Accept": "application/json",
        "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    },
    timeout=10,
)
response.raise_for_status()
payload = response.json()
print(len(payload["items"]))
print(payload["items"][0]["sku"])

Tested on 2 November 2026 against the local fixture:

5
A-1

The page that calls this endpoint is irrelevant to the crawl; the endpoint is the data source. Copy only the headers the server actually requires. Cookies, CSRF tokens, and Authorization headers are often session-specific; do not hard-code them.

GraphQL

A GraphQL request is usually a POST with a JSON body:

{
  "query": "query Items($first: Int!) { items(first: $first) { edges { node { id name price } } pageInfo { hasNextPage endCursor } } }",
  "variables": { "first": 100 }
}

Replay it with the same Content-Type: application/json and any required auth header:

import requests

query = """
query Items($first: Int!) {
  items(first: $first) {
    edges { node { id name price } }
    pageInfo { hasNextPage endCursor }
  }
}
"""

response = requests.post(
    "https://api.example.com/graphql",
    json={"query": query, "variables": {"first": 100}},
    headers={
        "Content-Type": "application/json",
        "Accept": "application/json",
        "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
        # "Authorization": "Bearer <token>",  # only if the API documents it
    },
    timeout=20,
)

This is illustrative; it is not run against a live GraphQL API. Read the schema and the API’s terms. GraphQL pagination is defined by the server: follow pageInfo.endCursor until hasNextPage is false.

What to copy, and what not to

Copy:

the URL and method;
the query parameters or JSON body;
the Accept and Content-Type headers;
the API version, if it is in the path;
pagination fields and sort parameters.

Do not copy blindly:

a session cookie from your browser and use it at scale;
an anti-bot token you do not understand;
an Authorization header you are not authorised to use;
a CSRF token that is tied to a session you do not own.

If the endpoint requires a login you do not have permission to use, stop. If it is a public endpoint with a documented rate limit, respect the limit. The sessions and APIs lesson covers the boundary.

GraphQL introspection

If the API allows introspection, you can ask for its schema:

{
  __schema {
    types {
      name
      kind
    }
  }
}

Many production APIs disable introspection. That is a deliberate choice, not a bug. Do not try to bypass it; read the public documentation instead.

When to use a browser instead

Use a browser when:

the data is assembled from many requests and no single response contains it;
the request is signed with a token generated at runtime and the API does not document a server-side flow;
you need to click, type, or scroll to trigger the data.

Even then, a browser is a rendering tool, not permission. The Playwright guide covers the mechanics.