Skip to content

Legal & ethics

Responsible crawling checklist: 12 questions before the first request

A practical pre-flight checklist for lawful, ethical crawling: purpose, robots.txt, terms, access controls, personal data, rate limits, retention, and stopping.

Last updated 3 min readBy CrawlerBros Editorial

Responsible crawling is a set of decisions made before the first request: what you need, whether you may take it, how much load you will impose, how long you will keep it, and how you will stop. This checklist is a practical summary of the legal and ethical material on this site. It is not legal advice.

Before you write code

  1. What is the purpose? Write one sentence. If it is “collect everything”, you do not have a purpose yet. A narrow purpose determines the fields, the retention, and the legal basis.
  2. Is the data public? Public does not mean unrestricted. It changes the risk profile, not the whole analysis.
  3. Is there an official API or dataset? Use it first. An API is a contract with documented fields and limits. A public dataset with a licence is better than a crawl.
  4. What do the terms say? Quote the clause on automated access, commercial use, resale, and personal data. If the terms prohibit the crawl, stop or ask for permission.
  5. Is there an access control? A login, paywall, CAPTCHA, API key, or technical block is a gate. Passing it without permission is high risk. Do not use a proxy or a browser to evade it.
  6. Does robots.txt allow the path? Parse the group for your user agent, apply longest match, and honour Crawl-delay as a request. The robots tester can check a rule.

Before you send traffic

  1. What is your rate limit? Per host, not per process. Start at one request per second for an unknown site, or the API’s documented limit minus a margin. Honour Retry-After on 429/503.
  2. Can you identify yourself? Use a descriptive user agent with a contact URL. Do not impersonate a browser or another crawler.
  3. What will you do on a block? Stop. Do not rotate IPs, change user agents, or retry a 403. A block is a decision, not a temporary failure.
  4. What personal data will you collect? If the answer is “some”, read the GDPR guide. Collect the minimum, document the basis, and plan deletion.

Before you store it

  1. How long will you keep it? Set a retention period for raw HTML, extracted records, logs, and personal data. Delete on a schedule.
  2. How will you stop and delete? Keep a kill switch, a list of sources, and a way to delete by source, record key, and person. If you cannot delete, you cannot honour a request or a legal hold.

Red flags

  • The plan requires a login you do not own.
  • The plan requires defeating a CAPTCHA or an anti-bot token.
  • The plan requires rotating IPs after a block.
  • The plan collects personal data at scale with no lawful basis.
  • The plan republishes copyrighted text or images without a licence.
  • The plan has no retention or deletion policy.
  • The only justification is “everyone does it” or “it is public”.

If any of these is true, stop and get advice. A successful crawl is not worth a legal claim or a breach of trust.

A short pre-flight record

Keep a one-page record per crawl:

Purpose:
Sources:
Official API checked: yes/no
Terms clause:
robots.txt result:
Access control: none/login/paywall/API key
Personal data: none/describe
Lawful basis (if personal data):
Rate limit:
User agent + contact:
Retention:
Deletion method:
Stop conditions:
Owner/date:

This record is useful for engineering, for a data-protection review, and for showing that you acted deliberately.

Sources

Sources checked