Blog
AI crawler policies in 2026: what the robots.txt data shows
A snapshot of AI crawler rules in the top 1,000 Tranco domains: 22.4% block GPTBot, 25.3% block CCBot, and only 2.7% block Googlebot. What the numbers do and do not mean.
In a 2 November 2026 sample of the top 1,000 Tranco domains, 474 served a valid robots.txt file. Of those, 22.4% disallowed GPTBot from the root, 22.2% disallowed ClaudeBot, 25.3% disallowed CCBot, 19.2% disallowed Google-Extended, and 11.8% disallowed OAI-SearchBot. Only 2.7% disallowed Googlebot.
The full dataset, method, and code are on the study page. This post is a short reading of the numbers, not a new study.
The headline is not “everyone blocks AI”
Most valid robots.txt files allow the AI crawlers at the root. Even for CCBot, the most-blocked agent in the sample, roughly three in four sites allow it. The story is a significant minority with a deliberate policy, not a universal wall.
The other headline is the gap between search and training:
| Agent | Root disallowed |
|---|---|
| Googlebot | 2.7% |
| OAI-SearchBot | 11.8% |
| PerplexityBot | 17.5% |
| GPTBot | 22.4% |
| ClaudeBot | 22.2% |
| CCBot | 25.3% |
| Google-Extended | 19.2% |
Sites are more willing to allow a search/answer crawler than a training crawler, but the difference is not clean. Some sites block both; some block training and allow search; some block everything.
The top of the list is more restrictive
The top 100 Tranco domains blocked GPTBot at 28.0%, compared with 15.9% in the 301–600 band. The middle bands are not uniformly less restrictive, and the top band has only 50 valid files. The pattern is suggestive, not conclusive: large platforms are more likely to have a written AI policy, a legal team, and a CDN that makes blocking easy.
Sitemaps are still the norm
72.8% of valid files declared a sitemap. That is a reminder that robots.txt is not only an access-control file; it is also a discovery file. A site can block a training crawler and still invite search crawlers to its sitemap.
Only 5.7% declared a Crawl-delay. The directive is not part of RFC 9309, and most sites rely on server-side rate limiting or a CDN instead.
What the numbers do not tell you
- They do not measure actual crawling. We fetched robots.txt only.
- They do not cover subdomains. A site’s API or mobile host may differ.
- They do not prove intent. A block can be a CMS default.
- They do not cover every AI crawler. New agents appear.
- They are a snapshot. Robots.txt changes; the date matters.
What to do with this
If you run a site, decide explicitly. A * group plus named groups is easier to audit than a pile of plugin defaults. Test the rules with the robots.txt tester. If you allow AI crawlers, say so; if you block training but allow search, say that too. Ambiguity is worse than a clear policy.
If you run a crawler, read the file for your user agent and respect it. A block is a decision, not a temporary failure.