Blog

CrawlerBros blog

Short, timely notes on crawler policy, library releases, and the work behind the site. Evergreen material lives in Learn, Guides, Compare, and Legal.

Timely notes on crawler policy, library releases, and the work behind the site. Evergreen material lives in the reference sections.

Last updated 2 November 2026

—AI crawler policies in 2026: what the robots.txt data showsA snapshot of AI crawler rules in the top 1,000 Tranco domains: 22.4% block GPTBot, 25.3% block CCBot, and only 2.7% block Googlebot. What the numbers do and do not mean.Updated 2 November 2026 · 3 min read

—Why CrawlerBros is now an independent crawling referenceThe old site sold scrapers. The catalogue is gone. Here is what we retired, what we kept, and why the new site has no products, no affiliate links, and no fake experts.Updated 2 November 2026 · 3 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.