Proxies and IP rotation for crawling: what they are, and when not to use them

A proxy is an intermediary that forwards an HTTP request, so the target sees the proxy’s address instead of the client’s. Proxies are legitimate infrastructure for caching, network policy, geo-testing, and separating services. They are not a permission system: rotating addresses to evade a site’s rate limit, ban, or access control is an ethical and often contractual problem.

The four common types

“Residential” and “mobile” proxies route through other people’s devices. Whether the exit node’s owner consented matters. A provider that does not explain its sourcing is a red flag. Using a proxy network to access data you are not permitted to access does not become lawful because the IP belongs to someone else.

What a proxy does not change

• Terms of service. The site’s rules apply to the request, not the IP.
• robots.txt. It is addressed to crawlers, not addresses.
• Copyright and database rights. The data is still the data.
• Privacy law. Personal data is still personal data.
• Access controls. A login, paywall, or CAPTCHA is still a boundary.

A proxy changes the network path. It does not change the authorisation.

Configure a proxy in requests

import requests

proxies = {
    "http": "http://user:password@proxy.example.com:8080",
    "https": "http://user:password@proxy.example.com:8080",
}

response = requests.get(
    "https://example.com/",
    proxies=proxies,
    headers={"User-Agent": "CrawlerBros-Example/1.0 (+https://www.crawlerbros.com/about)"},
    timeout=20,
)
print(response.status_code)

This is illustrative; it is not run against a live proxy. Keep proxy credentials in environment variables or a secret manager, not in source. A 407 Proxy Authentication Required means the proxy credentials are missing or wrong; it is not a target-site error.

Rotation is a rate-limit strategy, not a ban strategy

If you rotate IPs to avoid a per-IP rate limit, you are deliberately circumventing the site’s control. That can breach the terms and, depending on the access control and jurisdiction, the law. The defensible uses of rotation are:

• distributing load across your own egress pool for reliability;
• testing a site from multiple regions where the site permits it;
• accessing a site that explicitly offers geo-specific endpoints.

The indefensible use is “the site blocked me, so I will change addresses until it stops”. That is an arms race, and the site is telling you no.

Politeness with a proxy pool

If you operate a pool for a permitted crawl:

• keep a per-target rate limit, not just a per-IP limit;
• identify the crawler honestly in the user agent;
• do not rotate to evade 429 or 403;
• log which proxy served each request for debugging;
• remove failing proxies instead of retrying them blindly;
• never send credentials through a proxy you do not trust.

When to use a proxy

• Your service runs in a private network and needs a controlled egress.
• You need to test how a site behaves from another region, within its terms.
• You are caching at the edge and the proxy is part of the architecture.
• You have a contractual right to access the data and need a stable egress IP.

When not to use one

• To bypass a block, ban, paywall, login, or CAPTCHA.
• To hide a crawler’s identity while ignoring robots.txt.
• To access personal data at scale without a lawful basis.
• Because a tutorial said “use residential proxies” without explaining why.

The proxy types comparison covers the technical trade-offs. The terms of service guide covers the contractual side.

Sources

• RFC 9110 — HTTP Semantics
• RFC 1928 — SOCKS 5
• FTC — Protecting personal information