httpx vs requests vs aiohttp: choosing an HTTP client for crawling

requests is the default synchronous Python HTTP client. httpx is a modern client with sync and async APIs and HTTP/2 support. aiohttp is an async client/server framework built on asyncio. All three can fetch a page; they differ in concurrency model, API, and ecosystem.

At a glance

When to use requests

Use requests when:

• the crawl is small or medium and synchronous is enough;
• you want the largest ecosystem of examples and adapters;
• you need a simple Session with cookies and retries;
• you are writing a script, not a high-concurrency service.

It is the easiest client to debug and the one most tutorials assume. For a polite crawl at one request per second, the concurrency model is irrelevant.

When to use httpx

Use httpx when:

• you want one API for sync and async code;
• you need HTTP/2;
• you want a modern Client with explicit timeouts and connection limits;
• you are already using an async framework and do not want a second client.

import httpx

with httpx.Client(
    timeout=httpx.Timeout(10.0, connect=5.0),
    follow_redirects=True,
    headers={"User-Agent": "CrawlerBros-Example/1.0"},
) as client:
    response = client.get("https://example.com/")
    print(response.status_code, response.http_version)

This is illustrative. The important part is the explicit timeout object and the client reuse; creating a new client per request throws away the connection pool.

When to use aiohttp

Use aiohttp when:

• you are building an asyncio service and need a mature async client;
• you need a client and server in the same framework;
• you want fine-grained control over connectors and concurrency.

import asyncio
import aiohttp

async def main():
    timeout = aiohttp.ClientTimeout(total=10)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get("https://example.com/") as response:
            print(response.status, response.content_type)

asyncio.run(main())

Again illustrative. aiohttp is fast and capable, but the async model infects the rest of your code. Do not choose it just to “go faster” on a crawl that is limited by politeness.

Concurrency is not the same as speed

A synchronous client with one worker sends one request at a time. An async client can have many requests in flight, but if the site allows one request per second, the async client must still wait. Concurrency helps when latency dominates and the target permits parallelism; it does not override a rate limit.

Use a per-host limiter with any client. The rate limiting guide shows the pattern.

Timeouts and retries

• requests: timeout=(connect, read); retries via HTTPAdapter/urllib3 or a loop.
• httpx: httpx.Timeout(connect=..., read=..., write=..., pool=...); transport retries are available.
• aiohttp: ClientTimeout(total=..., connect=...); retries are usually a loop or a middleware.

Set a connect timeout and a read timeout separately. A connect timeout usually means the host is unreachable; a read timeout may mean a slow server. Do not set one infinite timeout and call it a day.

A practical recommendation

• Scripts and small crawls: requests.
• New sync or async Python code: httpx.
• Existing asyncio service: aiohttp.
• High-concurrency crawl: async client plus a per-host limiter, not just a bigger thread pool.

The HTTP client is rarely the reason a crawl fails. The failure is usually the target’s terms, a missing robots check, a rate limit, or a parser that broke. Choose the client that makes those behaviours easy to implement and test.

Sources

• requests documentation
• httpx documentation
• aiohttp documentation