Bot protection and Cloudflare: why a crawler gets blocked, and what to do

Bot protection is a set of controls a site uses to decide whether a request is allowed: rate limits, WAF rules, bot-management scores, JavaScript challenges, CAPTCHAs, and account/access controls. A block is a decision by the site. It is not a bug in your crawler, and it is not an invitation to evade.

This guide explains what you are seeing and the defensible responses. It does not teach bypasses.

What is probably happening

Symptom	Likely control	Defensible response
`403` with an HTML challenge page	WAF/bot management	Stop; read terms; use an API or ask for access
`429` with `Retry-After`	Rate limit	Slow down; honour the header
`503` with a Cloudflare/edge page	Edge protection or origin overload	Back off; do not retry in a tight loop
Empty HTML, then content after a delay	JavaScript challenge / client rendering	Find the JSON endpoint or use a browser only if permitted
CAPTCHA	Human-verification control	Do not solve it on behalf of others; use an API/permission or stop
Login wall	Authentication	Use an official API with your own credentials, or stop
`1020` / “access denied”	Firewall rule	Stop; the site has decided

A 403 on request 1 is often a header/WAF rule. A 403 after 300 requests is often a rate/IP decision. The fix is not the same, and neither is “rotate the IP”.

Diagnose before changing anything

Save the status, headers, and first 2 KB of the body.
Take a screenshot if a browser is involved.
Check robots.txt and the terms for the path.
Check whether an official API covers the data.
Check whether the block is per-IP, per-session, per-user-agent, or global.
Stop if the block is a policy decision.

A block page that says “automated access is not permitted” is not a technical puzzle. It is an answer.

What not to do

Do not rotate IPs to evade a block or a rate limit.
Do not solve CAPTCHAs on behalf of others.
Do not spoof a browser or another crawler’s user agent.
Do not reverse-engineer an anti-bot token to pass an access control.
Do not use a residential proxy network to hide a crawler that the site has blocked.
Do not keep retrying a 403; it will not become a 200.

These are not just ineffective; they can breach the terms and, depending on the access control and jurisdiction, the law. The terms guide and CFAA guide cover the legal side.

What to do instead

Use the official API. It is the path the site supports.
Ask for permission. A data licence or a partner agreement is normal for commercial use.
Slow down. A lower rate and a descriptive user agent can turn a block into a stable crawl.
Fix the request. Send the headers the API documents; do not send a browser’s full header set to an API.
Use a browser only when permitted. A browser executes JavaScript; it does not grant access.
Stop. If the site says no, the correct answer is no.

Cloudflare-specific notes

Cloudflare is a common edge provider, not a single control. A site may use:

a WAF rule that blocks a path or a user agent;
bot management that scores requests;
a managed challenge for suspicious traffic;
rate limiting;
a managed robots.txt with AI-crawler directives.

The response headers and body usually identify the edge. Read the site’s robots.txt and Cloudflare’s bot documentation. If the site publishes an AI-crawler policy, follow it. If it blocks your crawler, do not look for a “Cloudflare bypass”; look for an API or a permission.

A respectful crawler is not a stealth crawler

The signals of a respectful crawler are boring:

a descriptive user agent with a contact URL;
a per-host rate limit;
robots.txt compliance;
Retry-After compliance;
no login/paywall/CAPTCHA bypass;
a stop condition and a kill switch;
a retention and deletion policy.

If your crawler needs to hide those signals to work, the problem is the crawl, not the detection.