Sessions, cookies, and APIs: how state changes a crawl

A session is server-side state tied to a client by an identifier, usually a cookie. Authentication is how a client proves who it is; authorisation is what that identity may do. A crawler that keeps a session can fetch pages that depend on state; a crawler that uses an API can fetch data under a documented contract. Neither grants permission to access data you should not have.

How cookies work

A server sends Set-Cookie: session=abc123; Path=/; HttpOnly; Secure; SameSite=Lax. The client stores it and sends Cookie: session=abc123 on matching requests. The server looks up the session ID and remembers the user, the cart, the locale, or the CSRF token.

Cookie attributes that matter:

• Domain: which hosts receive the cookie.
• Path: which paths receive it.
• Expires / Max-Age: persistence.
• Secure: HTTPS only.
• HttpOnly: not readable by JavaScript.
• SameSite: whether cross-site requests carry it.

A crawler that needs a login must preserve the cookie jar across requests and handle expiry. A crawler that does not need a session should not keep cookies: they are identifiers, and hoarding them is a privacy and hygiene problem.

A session in requests

import requests

with requests.Session() as session:
    session.headers["User-Agent"] = "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"
    first = session.get("http://127.0.0.1:8765/session", timeout=10)
    second = session.get("http://127.0.0.1:8765/whoami", timeout=10)
    print(first.status_code, second.status_code)
    print(second.text.strip())

Tested on 2 November 2026 against a local fixture that sets a session cookie:

200 200
session=fixture

requests.Session keeps the cookie jar, connection pool, and default headers. The second request carries the cookie automatically. If you use httpx, the equivalent is httpx.Client().

When a session is not enough

Many sites pair a session cookie with a CSRF token. The token is embedded in a form or a meta tag and must be sent back in a header or a form field. A crawler that replays only the cookie gets a 403.

The pattern is:

1. GET the form or page.
2. Parse the CSRF token from the HTML or a cookie.
3. POST the form with the token and the other fields.
4. Keep the session cookie from the response.

This is normal for a documented API or a form you are permitted to use. It is not a technique for bypassing a login you do not own. If you find yourself reverse-engineering an anti-bot token to get past an access control, stop and use the official API or ask for permission.

Tokens and API keys

API authentication commonly uses one of:

• API key in a header (X-API-Key) or a query parameter.
• Bearer token in Authorization: Bearer <token>.
• OAuth 2.0 access token, often with a refresh token and scopes.
• Signed requests with a timestamp and HMAC.

Read the API documentation for the exact scheme, the scopes, the rate limits, and the terms. Store credentials in environment variables or a secret manager, never in a notebook, a repository, or a screenshot.

import os
import requests

token = os.environ["EXAMPLE_API_TOKEN"]
response = requests.get(
    "https://api.example.com/v1/items",
    headers={
        "Authorization": f"Bearer {token}",
        "Accept": "application/json",
        "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
    },
    params={"limit": 100},
    timeout=20,
)
response.raise_for_status()
payload = response.json()
print(payload["items"][0]["id"])

This example is illustrative; it is not run against a live third-party API. The shape is the point: a documented endpoint, a token from the environment, an explicit Accept, and a timeout.

API pagination

APIs paginate with one of four schemes:

A cursor or a next link is more stable than an offset: new records inserted at the front do not shift the page boundary. Stop when the response has no next cursor, an empty item list, or a repeated cursor. Do not trust “page 999” to be empty without checking the response.

Authentication is a boundary, not a puzzle

A login wall, a paywall, a private API, or a CAPTCHA is an access control. Using someone else’s session cookie, credential stuffing, or a token you found in a client bundle is not “scraping”; it is unauthorised access, and it can be a crime or a breach of contract depending on the facts.

The defensible paths are:

1. Use the official API with your own credentials.
2. Ask the site for permission or a data licence.
3. Scrape only public data, at a polite rate, within the terms.
4. Stop.

The terms of service guide and the CFAA guide explain the legal side. The responsible crawling checklist is the practical version.

Key takeaways

• HTTP is stateless; cookies and tokens add state.
• Keep a session only when the crawl needs one, and store credentials outside the code.
• A CSRF token is normal for a permitted form or API; bypassing an access control is not.
• Prefer cursor or next-link pagination over offsets.
• Authentication is not authorisation, and neither is permission to scrape.