Research

Reproducible web-crawling research

Original measurements with published methodology, datasets, and code. Every number is reproducible; every study carries a visible data-as-of date.

Original measurements, published with the dataset, the code, and the method. If you cannot reproduce a number, it should not be on the page.

Last updated 2 November 2026

—Research methodology: how the robots.txt study was measuredThe exact method behind the 2026 robots.txt study: sample, request policy, parser, valid-file definition, fields, limitations, and reproduction commands.Updated 2 November 2026 · 3 min read

—robots.txt in the top 1,000 Tranco domains: 2026 studyWe fetched robots.txt from the top 1,000 Tranco domains. Of 474 valid files, 22.4% block GPTBot, 25.3% block CCBot, and 72.8% declare a sitemap. Dataset and code published.Updated 2 November 2026 · 4 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.