Legal & ethics
GDPR and scraped personal data: lawful basis, transparency, and retention
When scraped data is personal data, which GDPR lawful basis can apply, what Article 14 transparency requires, and how CCPA/CPRA treats public information. Not legal advice.
The GDPR applies to the processing of personal data by an organisation in scope, regardless of whether the data was obtained by scraping, an API, or a purchase. “Publicly available” is not a blanket exemption. If a record identifies a person, it is likely personal data.
This page is research material, not legal advice. It summarises the GDPR and CCPA/CPRA; it does not apply them to your facts.
What counts as personal data
Under GDPR Article 4(1), personal data is any information relating to an identified or identifiable natural person. That includes:
- names, usernames, handles, profile URLs;
- email addresses, phone numbers, postal addresses;
- IP addresses and device identifiers in many contexts;
- photos, voice, and video;
- opinions, comments, and reviews tied to a person;
- inferred data (interests, behaviour, location).
A dataset that looks “anonymous” can become personal data when combined with other records. Pseudonymisation reduces risk; it does not remove the data from the GDPR’s scope.
Who is the controller?
The organisation that decides why and how the data is processed is the controller. A scraper that collects profiles for its own purposes is usually a controller. A service provider processing on someone else’s instructions may be a processor, with a written Article 28 agreement.
“We only scrape” is not a role analysis. Decide who determines the purposes and means, and document it.
Lawful basis (Article 6)
You need a lawful basis for each purpose. The common candidates for scraping:
| Basis | When it might apply | Problems |
|---|---|---|
| Legitimate interests (Art. 6(1)(f)) | Research, security, fraud prevention, some public-interest monitoring | Requires a balancing test; not available to public authorities for their public tasks |
| Consent (Art. 6(1)(a)) | You have valid, specific, informed consent from each person | Impractical for a large public crawl |
| Contract (Art. 6(1)(b)) | Processing necessary to perform a contract with the person | Rarely applies to a scraped third party |
| Legal obligation (Art. 6(1)(c)) | A specific legal duty | Narrow |
| Public task (Art. 6(1)(e)) | Public authorities acting in their mandate | Not for a private crawler |
Legitimate interests requires a three-part test: a legitimate interest, necessity, and a balancing of interests against the rights and freedoms of the people concerned. A commercial crawl of personal data at scale is harder to justify than a narrow research project with safeguards.
Special-category data (Article 9)
Data revealing racial or ethnic origin, political opinions, religious beliefs, health, sex life, sexual orientation, trade-union membership, genetic data, and biometric data for identification requires an Article 9 condition in addition to an Article 6 basis. Scraping a public page does not create one. Avoid collecting special-category data unless you have a clear legal basis and a strong need.
Transparency (Articles 13 and 14)
If you collect personal data directly from the person, Article 13 applies. If you obtain it from someone else (including a public website), Article 14 applies. You must provide information such as:
- the identity and contact details of the controller;
- the purposes and lawful basis;
- the categories of personal data;
- the recipients;
- retention periods;
- the person’s rights;
- the source of the data.
Article 14 has limited exceptions, but “the data was public” is not a general one. A privacy notice on your site is the usual way to meet this, but it must actually describe the scraping.
Rights and retention
People have rights to access, rectification, erasure, restriction, objection, and portability. You need a process to find and delete a person’s records. That is hard if you store raw HTML without an index by person.
Retention must be justified. “Keep everything forever” is not. Decide a period, document it, and delete on a schedule. The data storage guide covers the engineering side.
Security
Article 32 requires appropriate technical and organisational measures. For scraped data, that means access control, encryption at rest and in transit, logging, and a plan for a breach. A public dataset can still be a breach if it contains personal data and is exposed.
CCPA/CPRA
The California Consumer Privacy Act, as amended by the CPRA, gives California residents rights to know, delete, correct, and opt out of the sale or sharing of personal information. It also requires notice at collection and limits data use.
The CCPA has an exemption for “publicly available” information, but the definition is narrow: information lawfully made available from government records, or information the business has a reasonable basis to believe is lawfully made available to the general public by the consumer or widely distributed media. It is not a blanket exemption for anything a crawler can reach. Do not assume a public profile is outside the CCPA.
A practical checklist
- Does the dataset identify a person? If yes, treat it as personal data.
- What is the purpose, and is it necessary?
- Which Article 6 basis applies, and can you document the balancing test?
- Does it include special-category data? If yes, stop and get advice.
- How will you meet Article 14 transparency?
- How will you find and delete one person’s records?
- How long will you keep the data?
- Who can access it, and how is it secured?
- Is a DPIA required because of scale, sensitivity, or systematic monitoring?
- Have you checked the site’s terms and the local law?