Research
robots.txt in the top 1,000 Tranco domains: 2026 study
We fetched robots.txt from the top 1,000 Tranco domains. Of 474 valid files, 22.4% block GPTBot, 25.3% block CCBot, and 72.8% declare a sitemap. Dataset and code published.
We fetched https://<domain>/robots.txt for the top 1,000 domains in the Tranco list on 2 November 2026. Of the 474 domains that served a valid robots.txt, 22.4% disallowed GPTBot from the root, 25.3% disallowed CCBot, 19.2% disallowed Google-Extended, and 11.8% disallowed OAI-SearchBot. Only 2.7% disallowed Googlebot.
The dataset is published as CSV and JSON. The method, the script, and the limitations are below. If you cannot reproduce a number, do not cite it.
The sample
| Stage | Count |
|---|---|
| Domains attempted | 1,000 |
| HTTP 200 responses | 535 |
| Valid robots.txt files | 474 |
| 404/410 | 44 |
| 401/403 | 64 |
| 5xx | 5 |
| No HTTPS response (DNS/TLS) | 341 |
| 200 but HTML/JSON/empty (soft 404) | 61 |
| Truncated at 512 KB | 6 |
The Tranco top 1,000 is not a list of 1,000 content websites. It includes CDNs, DNS providers, ad networks, and domains that do not answer HTTPS at all. That is why 341 domains produced no HTTP response. The percentages below use the 474 valid robots.txt files, not the full 1,000, because a domain with no robots.txt cannot block or allow anything.
What the files say
| User agent | Disallowed from / |
Share of 474 |
|---|---|---|
* (all crawlers) |
35 | 7.4% |
| Googlebot | 13 | 2.7% |
| GPTBot | 106 | 22.4% |
| OAI-SearchBot | 56 | 11.8% |
| ClaudeBot | 105 | 22.2% |
| PerplexityBot | 83 | 17.5% |
| CCBot | 120 | 25.3% |
| Google-Extended | 91 | 19.2% |
Disallowed from / means the root path / is not allowed for that user agent after applying the group selection and longest-match rules. It is a strict root test: a site can block the root but allow specific paths, or block specific paths while allowing the root. The dataset records the root decision, not every path.
Training vs search
We grouped the agents into training (GPTBot, ClaudeBot, CCBot, Google-Extended) and search/answer (OAI-SearchBot, PerplexityBot):
| Pattern | Domains | Share of 474 |
|---|---|---|
| Blocks at least one training agent | 133 | 28.1% |
| Blocks at least one search/answer agent | 85 | 17.9% |
| Blocks both training and search | 84 | 17.7% |
| Allows all six | 339 | 71.5% |
The groups overlap: most sites that block search also block training. A site that blocks GPTBot but allows OAI-SearchBot is making a training/search distinction; a site that blocks both is treating the operator as a single policy question.
Rank bands
| Tranco rank | Valid files | GPTBot blocked | CCBot blocked | OAI-SearchBot blocked | Googlebot blocked | Sitemap declared |
|---|---|---|---|---|---|---|
| 1–100 | 50 | 28.0% | 30.0% | 22.0% | 4.0% | 72.0% |
| 101–300 | 84 | 23.8% | 28.6% | 11.9% | 3.6% | 69.0% |
| 301–600 | 138 | 15.9% | 18.8% | 6.5% | 0.7% | 73.9% |
| 601–1,000 | 201 | 24.9% | 27.4% | 12.9% | 3.5% | 73.6% |
The top 100 block AI crawlers at a higher rate than the middle bands, but the relationship is not monotonic. The sample is small in the top band (50 valid files), so treat the difference as a signal, not a precise estimate.
Sitemaps and crawl-delay
- 345 of 474 valid files (72.8%) declared at least one sitemap.
- 27 of 474 (5.7%) declared a
Crawl-delay. - The most common
Crawl-delayvalues were 10 seconds (10 sites), 1 second (6), and 2, 4, 5, or 0.1 seconds (2 each).
Crawl-delay is not part of RFC 9309. Its presence is a site asking for a slower crawl, not a standard rule. A crawler that ignores it is not violating the RFC, but it is ignoring a request.
What this does not tell you
- It does not measure actual crawling. We fetched robots.txt only. We did not fetch a page, follow a sitemap, or test a block.
- It does not cover subdomains. A site may have a different policy on
www,m, or an API host. - It does not cover every AI crawler. New user agents appear; we tested a fixed list.
- It does not prove intent. A
Disallowcan be a CMS default, a security plugin, or a deliberate policy. - It is a snapshot. Robots.txt changes. The dataset is dated 2 November 2026.
Why publish it
AI crawler policy is a live question for publishers, and most public claims are anecdotal. A dated, reproducible sample is more useful than a list of examples. The method is deliberately conservative: one request per domain, four concurrent workers, a 12-second timeout, a 512 KB cap, one attempt, and a user agent with a contact URL.
The next run should use the same script and a new date. If the numbers move, the change is the story.
Reproduce it
The script is scripts/relaunch/robots-study.mjs in this site’s repository. It reads the Tranco CSV and writes the dataset. The published CSV is the reclassified dataset; robots-analyze.mjs adds the valid column and recomputes the summary without refetching.
node scripts/relaunch/robots-study.mjs 1000
node scripts/relaunch/robots-analyze.mjs
The first command fetches 1,000 robots.txt files. Run it from a machine with a stable network and do not increase the concurrency without a reason. The second command is offline.