Terms of service and scraping: contracts, cease-and-desist, and practical risk

Terms of service are the contract a site offers for using it. Scraping can breach that contract even when the data is public and robots.txt allows the path. Whether the terms are enforceable, and what remedy follows, depends on how the terms were presented, what they prohibit, and the jurisdiction.

This page is research material, not legal advice. It summarises sources; it does not apply them to your facts.

Browsewrap, clickwrap, and sign-in terms

Courts distinguish how a user assented to terms:

• Clickwrap: the user clicks “I agree” or checks a box. Usually the strongest evidence of assent.
• Browsewrap: a link in the footer, with no affirmative action. Enforcement depends on whether a reasonable user had notice.
• Sign-in / API terms: accepted when creating an account or requesting a key. Often the most relevant terms for scraping, because they are tied to access.

A crawler that never creates an account may still be bound by browsewrap terms if the site presents them clearly and the crawler had notice. A crawler that uses an API is usually bound by the API terms explicitly.

What terms commonly prohibit

• automated access without permission;
• circumventing rate limits, access controls, or technical measures;
• using data for commercial purposes without a licence;
• reselling or redistributing data;
• creating a competing service;
• collecting personal data;
• using the site’s trademarks or content in a misleading way.

Not every prohibition is enforceable in every jurisdiction. A clause that purports to ban all access to public data may be challenged; a clause that protects an account, a private area, or a licensed API is more likely to be enforced.

What the cases say

hiQ Labs v. LinkedIn (9th Cir. 2022) held that scraping publicly available LinkedIn profiles was likely not “without authorization” under the CFAA. The court emphasised that the data was public and that the CFAA’s “without authorization” concept is about gates, not general terms. The case did not hold that scraping is always lawful; it addressed the CFAA question and remanded. The parties later settled, and the litigation did not produce a blanket rule.

Meta Platforms v. Bright Data (N.D. Cal. 2024) addressed Meta’s contract claims against a scraping company. The court granted summary judgment for Bright Data on the breach-of-contract claims at issue, in part because the terms did not clearly bind the conduct and the public data was not access-controlled. The case is a reminder that a terms clause is not self-executing; a court asks what the contract said and whether the conduct breached it.

Van Buren v. United States (2021) narrowed the CFAA’s “exceeds authorized access” language. It is a criminal case about a police database, not scraping, but it changed how courts read the CFAA’s scope. See the CFAA guide.

Cease-and-desist letters

A cease-and-desist letter can change the analysis. After a clear demand to stop, continuing to access a site may support a claim that access became “without authorization” or that the contract was breached. It can also be evidence of bad faith.

If you receive one:

1. Stop the crawl.
2. Preserve the logs and the terms you relied on.
3. Talk to a lawyer.
4. Do not rotate IPs or change user agents to continue.

A practical risk assessment

Ask:

1. Is the data public? Public does not mean unrestricted, but it affects the analysis.
2. Is there an access control? A login, paywall, CAPTCHA, or API key is a gate. Passing it without permission is high risk.
3. What do the terms say? Quote the clause. Does it prohibit automated access, commercial use, or resale?
4. How were the terms presented? Clickwrap, sign-in, or footer link.
5. Is there an official API or licence? Prefer it.
6. Does the site object? A cease-and-desist is a strong signal.
7. What is the jurisdiction? Contract, CFAA, state law, GDPR, and database rights differ.
8. What is the data? Personal data, copyrighted content, and databases raise separate issues.

What to do instead

• Use the official API and follow its terms.
• Ask for permission or a data licence.
• Use a public dataset with a clear licence.
• Crawl only public data, at a polite rate, within the terms.
• Stop when the site objects.

A proxy, a browser, or a rotating user agent does not change the contract. It may add evidence of evasion.

Sources

• hiQ v. LinkedIn, 9th Cir. 2022
• Meta v. Bright Data docket
• Van Buren v. United States