The robots.txt study measured one thing: whether a domain’s robots.txt disallows a given user agent from the root path. This page documents the method so the result can be checked, criticised, and re-run.

Sample

The sample is the first 1,000 rows of the Tranco top-1m daily list, downloaded on 2 November 2026. Tranco ranks domains by a combination of Alexa, Cisco Umbrella, Majestic, and Quantcast data. It is a convenience sample of popular domains, not a random sample of the web.

The list includes infrastructure domains (CDNs, DNS providers, ad networks) that do not serve a website. We did not filter them out before fetching; we report them as non-responses. Filtering them out after the fact would make the sample look cleaner but less honest.

Request policy

URL: https://<domain>/robots.txt only. No page fetches, no sitemap fetches, no subdomain discovery.
User agent: CrawlerBros-Research/1.0 (+https://www.crawlerbros.com/research/methodology; research contact: info@crawlerbros.com).
Concurrency: four workers.
Delay: 120–200 ms before each request, randomised.
Timeout: 12 seconds per request.
Redirects: followed by the HTTP client.
Retries: none. A failed domain is recorded as a failure.
Body cap: 512 KB; larger bodies are marked truncated and excluded from the valid set.
Headers: Accept: text/plain, */*;q=0.1, Accept-Encoding: gzip, deflate, br.

The run is intentionally conservative. It does not attempt to bypass a block, change user agents, or retry a failure. A domain that returns 403 is recorded as 403.

Valid-file definition

A response is a valid robots.txt file when:

the HTTP status is 200;
the body was not truncated at 512 KB;
the body is not empty;
the Content-Type is not text/html, application/json, an image type, or application/octet-stream;
the body does not start with <.

This excludes soft 404s that return 200 text/html with an error page. It does not exclude every possible non-robots response; a text/plain error page would pass. The limitation is stated rather than hidden.

Parsing and matching

The parser is a small implementation of the RFC 9309 group model:

Lines are split on newlines; # starts a comment.
User-agent starts a group; consecutive user-agent lines share the group.
Allow and Disallow add rules; empty values are ignored.
Crawl-delay is recorded but not used in the matching decision.
Sitemap lines are counted and the first is recorded.

For a target token, the parser selects the most specific matching group (exact token, case-insensitive), falling back to *. It then matches the path / against the rules:

* matches any sequence;
$ anchors the end;
the longest matching pattern wins;
Allow wins a tie;
no matching rule means allowed.

The study reports the root decision. It does not test a sample of deep paths. A site that allows / but disallows /articles/ is counted as allowing the root.

Fields in the dataset

Field	Meaning
`rank`	Tranco rank
`domain`	Domain as listed
`status`	HTTP status, or 0 for a network error
`contentType`	Response content type
`bytes`	Body size
`truncated`	Body exceeded 512 KB
`valid`	Passed the valid-file definition
`sitemapCount`	Number of `Sitemap:` lines
`crawlDelay`	Value for Googlebot, if any
`wildcardRoot`	Root allowed for `*`
`googlebotRoot`	Root allowed for Googlebot
`gptbot`	Root allowed for GPTBot
`oaiSearchbot`	Root allowed for OAI-SearchBot
`claudeBot`	Root allowed for ClaudeBot
`perplexityBot`	Root allowed for PerplexityBot
`ccbot`	Root allowed for CCBot
`googleExtended`	Root allowed for Google-Extended
`error`	Error class for a failed request

Limitations

Popularity sample, not random. The result describes the Tranco top 1,000, not the web.
Root only. Deep-path rules are not measured.
One snapshot. Robots.txt changes; the date is part of the result.
Fixed user-agent list. New crawlers are not included.
No intent. A rule can come from a CMS default, a security plugin, or a deliberate policy.
Network failures. 341 domains produced no HTTPS response. The valid-file denominator is 474.
Content-Type heuristic. A text/plain error page can pass as a valid file.

Reproduction

node scripts/relaunch/robots-study.mjs 1000
node scripts/relaunch/robots-analyze.mjs

The first command fetches the sample and writes reports/relaunch/robots-2026.csv and reports/relaunch/research/robots-2026.json. The second reclassifies an existing CSV and recomputes the summary without network access.

If you re-run it, change the date and keep the old dataset. A study that overwrites its history cannot show change.

Research methodology: how the robots.txt study was measured