Skip to content

About

Terms of use

The terms for using CrawlerBros: content licence, code licence, acceptable use, no warranty, external links, and why nothing here is legal advice.

Last updated 3 min readBy CrawlerBros Editorial

These terms apply to crawlerbros.com. By using the site, you agree to them. If you do not agree, do not use the site.

What this site is

CrawlerBros is an independent editorial reference about web crawling and web scraping. It is not a law firm, a data broker, a hosting provider, or a scraping service. Nothing on the site is legal advice, and no page creates a lawyer-client relationship.

Content licence

Unless a page says otherwise, the text of CrawlerBros pages is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You may share and adapt it for any purpose, including commercially, if you give appropriate credit, link to the licence, and indicate whether changes were made.

A practical attribution is:

Source: CrawlerBros, “Page title,” last updated 2 November 2026, https://www.crawlerbros.com/….

Code samples are licensed under the MIT licence. You may use, copy, modify, and distribute them, including in commercial software, provided the copyright notice and permission notice are included. The code is provided as-is, without warranty.

Third-party quotations, logos, and linked material remain the property of their owners and are not covered by these licences. If you quote a source that we quote, cite the original source.

Acceptable use

You may read, link to, quote, and crawl this site. If you crawl it, identify your bot, respect robots.txt, use a reasonable request rate, and do not attempt to access non-public areas. The site is static; there are no accounts to attack.

You may not:

  • use the site to break the law or to infringe someone else’s rights;
  • misrepresent the site or its content as legal advice;
  • remove attribution from text you republish under CC BY 4.0;
  • use the CrawlerBros name or logo in a way that suggests endorsement, partnership, or sponsorship without written permission;
  • interfere with the site’s availability or security.

No warranty

The site is provided “as is” and “as available.” We work to keep it accurate and current, but we do not warrant that every page is complete, error-free, or suitable for your purpose. Library versions change, websites change, and the law changes. A code sample that worked on the date shown may need changes later.

To the maximum extent permitted by law, CrawlerBros and its operator are not liable for any indirect, incidental, special, consequential, or punitive damages, or for any loss of data, profits, or business opportunity, arising from the use of the site or reliance on its content. Nothing in these terms limits liability that cannot be limited by law.

The site links to RFCs, statutes, court opinions, documentation, and vendor pages. We do not control those sites and are not responsible for their content, availability, or privacy practices. A link is not an endorsement, and a vendor link is not a paid placement.

Comparisons and opinions

Comparisons state their criteria and dates. They are editorial opinions based on the evidence cited. A tool or service may change after the date of a comparison; check the current documentation and pricing before relying on it.

Crawling this site

Crawling is allowed subject to robots.txt. The site publishes a sitemap and an Atom feed. If you operate a crawler, use a descriptive user agent with a contact URL, honour Crawl-delay if present, and do not hammer the origin. We may rate-limit abusive traffic.

Changes

We may update these terms. The last updated date at the top changes when we do. Continued use after a change means you accept the updated terms.

Contact

Questions about these terms: info@crawlerbros.com.