Async Python scraping with httpx: bounded concurrency without chaos

Async scraping lets one process keep many requests in flight while waiting on the network. It does not make a site tolerate more requests. The useful pattern is bounded concurrency plus a per-host rate limit: overlap the waiting, cap the load.

httpx.AsyncClient is the modern choice when you want one API for sync and async code, HTTP/2 support, and explicit connection limits.

Install

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\Activate.ps1
python -m pip install httpx

Tested with Python 3.14.2 and httpx 0.28.1 on 2 November 2026.

Fetch a batch with a client

import asyncio
import httpx


async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    response = await client.get(url)
    response.raise_for_status()
    return response.json()


async def main() -> list[dict]:
    async with httpx.AsyncClient(
        headers={
            "User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)",
            "Accept": "application/json",
        },
        timeout=10,
        limits=httpx.Limits(max_connections=5, max_keepalive_connections=5),
    ) as client:
        pages = await asyncio.gather(
            *(fetch(client, f"http://127.0.0.1:8765/api/items?page={number}") for number in range(1, 4))
        )
    return pages


pages = asyncio.run(main())
print(len(pages), [len(page["items"]) for page in pages])

Tested on 2 November 2026 against the local fixture:

3 [5, 5, 5]

The fixture returns the same five items for each page; the point is the client reuse and the bounded connection pool, not the fixture’s pagination.

Bound the concurrency

asyncio.gather with 1,000 coroutines opens 1,000 requests at once unless the client limits them. max_connections caps the pool, but you should also cap the number of tasks you create:

import asyncio
import httpx

semaphore = asyncio.Semaphore(5)

async def fetch(client: httpx.AsyncClient, url: str) -> int:
    async with semaphore:
        response = await client.get(url)
        response.raise_for_status()
        return response.status_code

async def main(urls: list[str]) -> list[int]:
    async with httpx.AsyncClient(timeout=10, limits=httpx.Limits(max_connections=5)) as client:
        return await asyncio.gather(*(fetch(client, url) for url in urls))

print(asyncio.run(main(["http://127.0.0.1:8765/"] * 10))[:3])

A semaphore is not a rate limit. It caps in-flight work; it does not stop five workers from firing five requests at the same instant. Add a per-host limiter (token bucket or next-allowed timestamp) for politeness.

Timeouts

httpx.Timeout lets you set connect, read, write, and pool timeouts separately:

timeout = httpx.Timeout(10.0, connect=5.0)

A connect timeout means the host is unreachable; a read timeout means the server is slow. Retry the first with backoff; retry the second only with a cap. Do not set one infinite timeout and call it resilient.

Retries and status codes

httpx does not retry by default. Wrap the request in your own policy:

retry connection errors, timeouts, 429, and most 5xx;
do not retry 400, 401, 403, or 404 unchanged;
honour Retry-After on 429/503;
cap attempts and total time.

The retries guide has the policy and a tested loop.

When async is not worth it

A few dozen pages. Sync requests is simpler and fast enough.
A single host with a 1 req/s limit. Async does not help; the rate limit dominates.
A parser-heavy CPU workload. Async I/O will not speed up BeautifulSoup; use processes or a faster parser.
A team unfamiliar with asyncio. The debugging cost can exceed the throughput gain.

Use async when latency dominates and the target permits concurrency. Measure the target’s tolerance before raising the limit.