API-first scraping: when the endpoint is better than the page

API-first scraping means checking for a documented endpoint before parsing HTML. An API is a contract: it defines fields, authentication, pagination, rate limits, and terms. When it exists and covers the data you need, it is usually faster, more stable, and more defensible than a page scraper.

The hard part is not the HTTP call. It is deciding whether the API is allowed, what it actually returns, and how to behave when it says no.

1. Find the API before the parser

Look for:

a /docs, /api, /developers, or /api-docs link in the footer;
an OpenAPI/Swagger file (/openapi.json, /swagger.json);
a developer portal or a “Data API” page;
network requests in DevTools that return JSON, not only an internal endpoint;
a public dataset with a licence.

An internal JSON endpoint is not the same as a public API. The fact that a page calls it does not mean you may call it at scale. Read the terms and the documentation.

2. Read the contract

Before writing code, answer:

What is the base URL and version?
Which auth scheme: API key, bearer token, OAuth, signed request?
What are the scopes and the rate limits?
How does pagination work: offset, page, cursor, or Link header?
What are the required headers (Accept, Content-Type, User-Agent)?
What does an error look like (4xx body, error code, retry hint)?
What are the storage/display/attribution rules?
What is the retention or deletion obligation?

If the docs do not answer these, the API is not ready for a production crawl.

3. Make one request

import requests

response = requests.get(
    "http://127.0.0.1:8765/api/items",
    headers={
        "Accept": "application/json",
        "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    },
    timeout=10,
)
response.raise_for_status()
payload = response.json()
print(len(payload["items"]))
for item in payload["items"]:
    print(item["sku"], item["name"], item["price"])

Tested on 2 November 2026 against the local fixture:

5
A-1 Mechanical keyboard $89.00
B-2 USB-C hub $39.50
C-3 Laptop stand $54.00
D-4 Noise-cancelling headphones $199.00
E-5 Webcam $69.00

raise_for_status() is not optional. An API error body often looks like JSON; without the status check, you will parse an error object as data.

4. Authenticate without leaking

Keep credentials in environment variables or a secret manager:

import os
import requests

token = os.environ["EXAMPLE_API_TOKEN"]
response = requests.get(
    "https://api.example.com/v1/items",
    headers={"Authorization": f"Bearer {token}", "Accept": "application/json"},
    timeout=20,
)

This is the documented shape, not a run against a third-party API. Never commit a token, print it in logs, or put it in a client-side bundle. Use the narrowest scope and rotate it if it leaks.

5. Paginate to the end

Follow the API’s mechanism:

Scheme	Continue with	Stop when
Offset/limit	`offset += limit`	Short page or empty items
Page number	`page += 1`	Empty page or documented last page
Cursor	`cursor = response["next_cursor"]`	Null cursor or repeated cursor
`Link` header	the `rel="next"` URL	No `next` link

Cursor pagination is the most stable for changing datasets. Offsets shift when records are inserted at the front.

6. Respect the limits

Read the documented rate limit; stay under it.
Honour 429 and Retry-After.
Cache responses; do not re-fetch the same page in a loop.
Use conditional requests (ETag, Last-Modified) when supported.
Log the remaining quota if the API exposes it.

The API is a shared resource. A crawler that ignores its limits will be blocked, and the block will be deserved.

7. Validate the schema

Do not trust that a field is always present or always the same type. Validate the fields you depend on:

for item in payload["items"]:
    assert isinstance(item.get("sku"), str) and item["sku"]
    assert isinstance(item.get("price"), str) and item["price"].startswith("$")

In production, use a schema (Pydantic, JSON Schema, or a database constraint) and quarantine invalid records instead of writing nulls.

8. When an API is not the answer

The API does not cover the data. A public page may be the only source.
The API is private/internal. A page’s internal endpoint is not a public API.
The terms prohibit your use. An API key does not grant every use.
The API is stale or incomplete. Check the lastmod/updated_at and the docs.
The volume is tiny. A one-off page fetch may be simpler than an API integration.

The decision rule

Use the API when it is documented, permitted, and covers the fields you need. Use a page parser when the data is public, the terms allow it, and no API exists. Do not use a browser or a proxy to turn an internal endpoint into a public one.

The sessions and APIs lesson covers auth and state; the XHR/GraphQL guide covers finding the endpoint a page calls; the retries guide covers what to do when it fails.