Pagination and infinite scroll: find the real next page

Pagination splits a result set across multiple responses. The reliable way to crawl it is to follow the site’s own next-page mechanism: a rel="next" link, a cursor from the response, or a documented API parameter. Guessing ?page=2 works only when the site actually uses that scheme.

The four schemes

Cursors are the most stable for changing datasets. Offsets shift when records are inserted at the front; page numbers shift when the page size changes.

Follow the next link

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "http://127.0.0.1:8765"
HEADERS = {"User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"}
url = f"{BASE}/items?page=1"
records = []
seen_pages = set()

while url and url not in seen_pages:
    seen_pages.add(url)
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")
    for card in soup.select("ul.results li.item"):
        records.append({
            "sku": card["data-sku"],
            "name": card.select_one("h3").get_text(strip=True),
            "price": card.select_one(".price").get_text(strip=True),
        })
    next_link = soup.select_one('a[rel="next"]')
    url = urljoin(BASE, next_link["href"]) if next_link else None

print(len(records), [record["sku"] for record in records])

Output against the fixture:

5 ['A-1', 'B-2', 'C-3', 'D-4', 'E-5']

seen_pages prevents a loop when a site’s next link points back to the current page. The loop stops when url becomes None.

Cursor pagination in an API

import requests

BASE = "https://api.example.com/v1/items"
cursor = None
items = []

while True:
    params = {"limit": 100}
    if cursor:
        params["cursor"] = cursor
    response = requests.get(
        BASE,
        params=params,
        headers={"Authorization": "Bearer <token>", "Accept": "application/json"},
        timeout=20,
    )
    response.raise_for_status()
    payload = response.json()
    items.extend(payload["items"])
    cursor = payload.get("next_cursor")
    if not cursor:
        break

print(len(items))

This is illustrative, not run against a live third-party API. The shape is the point: send the cursor, append the page, stop when the cursor is absent. Keep a set of seen cursors if the API can repeat one.

Infinite scroll

Infinite scroll is usually a normal request triggered when the viewport nears the bottom. Before using a browser:

1. Open DevTools → Network → Fetch/XHR.
2. Scroll until the next batch loads.
3. Find the request that returned the new records.
4. Copy its URL, method, headers, and parameters.
5. Replay it with requests; increase the cursor/offset until the response is empty.

If the request needs a signature or a token that changes per session, use a browser or the official API. Do not reverse-engineer an anti-bot token to get past an access control.

A browser fallback:

from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://127.0.0.1:8765/items?page=1", wait_until="domcontentloaded")
    previous = 0
    while True:
        cards = page.locator("ul.results li.item")
        count = cards.count()
        if count == previous:
            break
        previous = count
        if page.locator('a[rel="next"]').count() == 0:
            break
        page.locator('a[rel="next"]').click()
        page.wait_for_load_state("domcontentloaded")
    print(previous)
    browser.close()

Output against the fixture:

2

The fixture replaces the page rather than appending, so the count stays at 2. For a true infinite-scroll page, compare the total item count after each scroll and stop when it stops growing.

Stop conditions that matter

• No next link/cursor. The normal end.
• Repeated page hash. Hash the record set; if it has not changed, break.
• Repeated cursor/URL. Keep a set; break on a repeat.
• Empty page. Stop, but verify it is genuinely empty and not a soft 404.
• Budget. A maximum page count or time budget prevents a runaway.
• Error threshold. Stop after N consecutive failures; do not hammer a failing site.

Common mistakes

• Guessing page numbers. A site may use ?p=2, ?page=2, ?start=20, or a cursor. Follow the real link.
• Ignoring the last page. Some sites return a 200 with an empty list; others return a 404. Handle both.
• Assuming stable order. New records can shift offsets. Use cursors or deduplicate by ID.
• Not deduplicating across pages. A record can appear on two pages during an update. Deduplicate by a stable key.
• Forgetting the delay. Pagination is still crawling; apply the same per-host rate limit.

Sources

• RFC 8288 — Web Linking
• Google Search Central — Pagination