Legal & ethics

Legal and ethical web crawling

What robots.txt, terms of service, the CFAA, GDPR, copyright, and AI-crawler policies actually say, with primary sources and a practical respectful-crawling checklist.

What the rules actually say, with links to the RFC, the statute, the court opinion, or the policy page. This is research material, not legal advice.

Last updated 2 November 2026

—robots.txt and RFC 9309: what the standard actually saysHow robots.txt matching works under RFC 9309: groups, Allow and Disallow, wildcards, longest match, crawl-delay, sitemaps, and the limits of the protocol.Updated 2 November 2026 · 4 min read

—Terms of service and scraping: contracts, cease-and-desist, and practical riskHow website terms interact with scraping, why browsewrap and clickwrap differ, what hiQ and Meta v. Bright Data decided, and how to assess contractual risk.Updated 2 November 2026 · 4 min read

—The CFAA and web scraping: Van Buren, hiQ, and the limits of 'unauthorized access'What the Computer Fraud and Abuse Act covers, how Van Buren narrowed 'exceeds authorized access', how hiQ applied it to public scraping, and why a cease-and-desist matters.Updated 2 November 2026 · 4 min read

—GDPR and scraped personal data: lawful basis, transparency, and retentionWhen scraped data is personal data, which GDPR lawful basis can apply, what Article 14 transparency requires, and how CCPA/CPRA treats public information. Not legal advice.Updated 2 November 2026 · 5 min read

—Copyright and database rights for scraped contentFacts are not copyrightable, but text, photos, and original selections can be. Learn the EU database right, the Feist standard, and how to assess what a crawl may reuse.Updated 2 November 2026 · 3 min read

—AI crawler policies: GPTBot, ClaudeBot, CCBot, and Google-ExtendedWhat the major AI crawler user agents control, how robots.txt directives apply, and how to write an explicit allow or block policy without breaking search.Updated 2 November 2026 · 3 min read

—Responsible crawling checklist: 12 questions before the first requestA practical pre-flight checklist for lawful, ethical crawling: purpose, robots.txt, terms, access controls, personal data, rate limits, retention, and stopping.Updated 2 November 2026 · 3 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.