Blog
Why CrawlerBros is now an independent crawling reference
The old site sold scrapers. The catalogue is gone. Here is what we retired, what we kept, and why the new site has no products, no affiliate links, and no fake experts.
CrawlerBros used to be the marketing site for a catalogue of ready-made scrapers. That business was discontinued in October 2026. The domain is now an independent reference for web crawling and web scraping: tested code, primary sources, reproducible measurements, and no products for sale.
What we retired
The old site had about 1,850 indexable URLs: 1,344 Actor pages, 15 suites, 15 capability hubs, 455 articles, and a handful of indexes. Almost every one existed to promote a scraper. When the catalogue went away, those pages had no reason to exist.
We retired them. A URL with a sensible neutral successor now returns a single-hop 301 to that page; the rest return a 404. We did not redirect everything to the homepage, because a mass redirect is a soft 404 with extra steps. The inventory and the mapping are in the relaunch audit in the repository.
What we kept
The domain, the brand name, and the design system. The name has history and links; the meaning changes from “a scraper vendor” to “an independent crawling resource”. The design system was already a good fit for technical material: a paper ground, a datasheet layout, and a cyan accent that is used as a fill, not as text on white.
We also kept the parts of the engineering that made the old site fast: prerendered HTML, a small client bundle, self-hosted fonts, and a strict distinction between server-only and client code.
What is new
- A six-lesson course that starts with HTTP and ends with sessions and APIs.
- Tested guides for Python, JavaScript, headless browsers, pagination, retries, sitemaps, incremental crawls, deduplication, storage, scheduling, and proxies.
- Vendor-neutral comparisons with dated facts and stated criteria.
- A legal and ethics section that cites the RFCs, statutes, and cases instead of repeating folklore.
- Seven free, client-side tools.
- A reproducible study of robots.txt across the top 1,000 Tranco domains.
- A glossary with thirty definitions.
What we will not do
We will not sell scrapers, take affiliate commissions, or accept paid placements. We will not invent authors, ratings, reviews, or case studies. We will not publish instructions for bypassing a login, paywall, CAPTCHA, or ban. We will not mass-generate thin pages to chase keywords.
If the funding model changes, the change will be disclosed on the page and in the editorial policy before it takes effect.
How to read the site
Start with Learn if you are new. Use Guides when you are building. Read Legal & ethics before you point a crawler at someone else’s site. Use the glossary when a term is unfamiliar. If you find an error, the corrections address is open.
The old site’s traffic was mostly brand queries and long-tail “X scraper” queries with almost no clicks. We are not trying to recreate that. We are trying to be the page a developer bookmarks and an answer engine cites because it is correct, dated, and reproducible.