About CrawlerBros

CrawlerBros is an independent editorial site about web crawling and web scraping. It publishes tested code, explains the rules with primary sources, and measures what it can reproduce. It does not sell scrapers, hosted crawling, consultancy, or affiliate placements.

What changed, and why

The domain previously hosted a product site for a publisher of ready-made scrapers. That business was discontinued in October 2026. The old site had 455 articles and about 1,850 indexable URLs, almost all of which existed to promote a catalogue of scrapers. When the catalogue went away, so did the reason for those pages.

Rather than leave a thin product archive online, we retired it. Every URL that had a sensible neutral successor now redirects to one; the rest return a clear 404. The inventory, redirect map, and the reasons are in the relaunch audit that ships with the repository.

The name stays because the domain has history and links. The meaning changes: CrawlerBros is now a reference, not a vendor.

What we publish

• Learn — six ordered lessons on HTTP, HTML, selectors, rendering, and sessions.
• Guides — tested Python and JavaScript walkthroughs, plus operational guides for pagination, retries, sitemaps, storage, scheduling, and proxies.
• Compare — dated, vendor-neutral comparisons with stated criteria.
• Legal & ethics — robots.txt, terms of service, the CFAA, GDPR, copyright, and AI-crawler policies, with links to the actual texts.
• Tools — free, client-side tools that require no account.
• Research — original measurements with the dataset, method, and code published.
• Glossary — short, quotable definitions that link into the longer material.

Who writes it

Pages are published under the byline CrawlerBros Editorial. Drafting is assisted by software, including language models; the editorial process is not. Every code sample is run against a local fixture before publication, every factual claim is checked against the source listed on the page, and every page carries the date it was last changed.

We do not invent authors, reviewers, testimonials, ratings, or case studies. There are no stock-photo experts on this site.

How we stay independent

There are no products for sale, no affiliate links, no sponsored posts, and no paid placements. If that changes, the change will be disclosed on the page and in the editorial policy before it takes effect. The site is funded by its owner; it is not funded by vendors whose tools it compares.

Corrections

If a page is wrong, tell us. The corrections address is corrections@crawlerbros.com. We log substantive corrections with the date and what changed. See the corrections log and the editorial policy.

Contact

General enquiries: info@crawlerbros.com. Research and data questions: use the same address and put “research” in the subject line. We do not provide legal advice or scrape sites on request.