Extracting Personal Data from Publicly Available Sources

Extracting Personal Data from Publicly Available Sources (CIPT): Data Collection, Use, Dissemination and Destruction

Overview
Extracting personal data from publicly available sources is a core topic in the IAPP CIPT (Certified Information Privacy Technologist) body of knowledge. It falls under the domain of data collection, use, dissemination and destruction. It covers how organizations and technologists gather personal information that is openly accessible, such as websites, social media, public registries, court records, news articles and open datasets. It also covers the privacy, legal, ethical and technical issues that come with doing so.

The central lesson is simple. Publicly available does not mean free to use for any purpose. Personal data keeps its status as personal data even when anyone can see it, and privacy obligations still apply.

Why It Is Important
1. Scale and automation: Web scraping, crawlers, APIs and AI training pipelines can collect huge volumes of personal data in minutes. The resulting aggregated profiles can be far more invasive than any single public fact.

2. Regulatory exposure: Regulators have taken enforcement action and imposed large fines over scraping-based practices.

• Clearview AI was fined by regulators in Italy, France, Greece, the UK and the Netherlands for scraping facial images.
• Global privacy authorities issued a joint statement on data scraping in 2023, with a follow-up in 2024.

3. Contextual integrity: People share information in a specific context, such as a professional profile shared for networking. Moving it into a different context, such as marketing, profiling or surveillance, can violate reasonable expectations.

4. Aggregation and inference harms: In Solove's taxonomy of privacy, aggregation is an information processing harm. Combining harmless public facts can reveal sensitive details such as location patterns, health, sexual orientation or political views. It can also enable stalking, doxxing, identity theft and social engineering.

5. Trust and reputation: Organizations seen as harvesting people's data without their knowledge suffer reputational damage, even where the activity was technically lawful.

6. Role of the technologist: Privacy technologists build and govern the systems that collect data. They are expected to embed privacy by design into scraping, crawling and data enrichment projects.

What It Is
The topic covers collecting personal information from sources with no access restriction, or minimal restriction. These sources include:

• Public websites and online forums
• Social media profiles and posts
• Government and public records: property, court, corporate and electoral registers
• Professional directories and business listings
• News media and archives
• Open data portals and leaked or breached datasets (the latter raise serious legal and ethical concerns)
• Images and video posted online

Common collection methods include:

• Web scraping: automated extraction of content from web pages.
• Web crawling or spidering: systematically following links to index content.
• APIs: structured access provided by platforms, usually governed by terms of service.
• Manual collection: human researchers gathering open-source intelligence (OSINT).
• Data brokers: buying compiled datasets built from public and other sources.

Legal Landscape
Treatment of public data differs by jurisdiction. This is a frequent exam point.

EU and UK GDPR

• Publicly available personal data is still fully protected. There is no general exemption.
• Collection still requires a lawful basis under Article 6. This is often legitimate interests, which requires a documented balancing test (legitimate interests assessment).
• Where special category data is involved, an Article 9 condition must also apply. Article 9(2)(e) covers data manifestly made public by the data subject, and it is interpreted narrowly.
• Transparency obligations still apply. Article 14 covers data not obtained from the data subject: individuals must be informed within a reasonable period, at most one month. The disproportionate effort exception exists but is construed narrowly.
• Data minimization, purpose limitation, accuracy, storage limitation and data subject rights all continue to apply.
• A DPIA is often required for large-scale or innovative processing, especially profiling or biometrics.

United States

• The US has a sectoral approach. Many state privacy laws, such as the CCPA/CPRA, exclude publicly available information from the definition of personal information.
• The definition is narrow. It typically covers information lawfully made available from government records, or information the consumer has made available to the general public without restriction. CPRA also excludes biometric information collected without the consumer's knowledge.
• Data broker registration laws exist in California, Vermont, Texas and Oregon. California's Delete Act creates a centralized deletion mechanism.
• The Computer Fraud and Abuse Act (CFAA) is relevant. In hiQ Labs v. LinkedIn, the court suggested that scraping publicly accessible data likely does not count as access 'without authorization' under the CFAA. However, hiQ was ultimately found to have breached LinkedIn's user agreement.
• Illinois BIPA restricts the collection of biometric identifiers, such as faceprints, even from public images.
• FTC Act Section 5 covers unfair or deceptive practices.
• The FCRA applies if scraped data is used for eligibility decisions such as employment, credit or housing.

Canada (PIPEDA)

• The 'publicly available' exception is narrow. It is limited to specific categories listed in regulation, such as telephone directories, professional directories, registries and some publications.
• Social media is generally not covered. The Clearview AI finding confirmed this.

Other jurisdictions
Australia, Singapore, Brazil (LGPD) and others generally treat public data as still protected, with some limited exceptions. Regulators repeatedly state that scraping can amount to a reportable data breach for the platform being scraped.

Contractual and technical restrictions
Website terms of service, robots.txt directives, rate limits, CAPTCHAs and API licensing terms can all restrict extraction. Ignoring them creates contract, tort and possibly criminal risk.

How It Works: Lifecycle and Controls
1. Define purpose and necessity
Document a specific, legitimate purpose. Ask whether personal data is actually needed, or whether aggregated or anonymized data would do.

2. Assess the source

• Is the data truly public, or is it behind a login or paywall?
• Did the individual choose to make it public, or was it published by a third party or leaked?
• What do the terms of service and robots.txt allow?
• Is the data reliable and accurate?

3. Establish a legal basis and conduct assessments
Carry out a Privacy Impact Assessment or DPIA and, where relying on legitimate interests, a legitimate interests assessment. Consider the reasonable expectations of data subjects, the sensitivity of the data, the potential for harm and whether vulnerable groups such as children are involved.

4. Apply privacy by design in the collection tool

• Minimization at the point of collection: configure scrapers to collect only the fields needed. Use filters to exclude special category data, children's data and unnecessary identifiers.
• De-identification: pseudonymize, hash, tokenize or aggregate as early as possible.
• Respect technical signals: honor robots.txt, rate limits and opt-out mechanisms.
• Provenance tracking: record the source, the date collected and the conditions of use as metadata.

5. Provide transparency and enable rights
Give notice through privacy policies, direct notice under Article 14 where feasible, or public-facing notices. Support access, deletion, objection and opt-out requests. Allow suppression lists so deleted individuals are not re-collected.

6. Secure and govern use

• Apply access controls, encryption and logging.
• Prevent function creep, meaning reuse for incompatible purposes.
• Control onward sharing and sale.
• Assess AI training use for memorization and regurgitation risks.

7. Retention and destruction
Set retention limits tied to the purpose. Use secure deletion methods, refresh data regularly to maintain accuracy, and purge data that is stale or was collected unlawfully.

Platform Perspective: Defending Against Scraping
The CIPT also expects you to understand the other side. Platforms hosting personal data have security and privacy obligations to protect users from unauthorized scraping. Regulators expect a multi-layered set of controls:

• Rate limiting and throttling
• Bot detection and CAPTCHAs
• Monitoring for unusual access patterns
• Restricting what is visible to logged-out users
• Privacy-protective default settings
• Legal action and terms of service enforcement
• User education about visibility settings

Mass scraping may count as a data breach requiring notification in some jurisdictions.

Key Concepts and Frameworks to Remember

• Contextual integrity (Nissenbaum): information flows should match the norms of the context in which the data was shared.
• Solove's taxonomy: aggregation, identification, secondary use, exclusion and increased accessibility are all relevant harms.
• FIPPs / OECD principles: collection limitation, purpose specification, use limitation, openness and individual participation.
• Privacy by design: proactive, embedded and default protections.
• OSINT: open-source intelligence, often used in security, investigations and threat intelligence. It still requires a legal basis and proportionality.
• Re-identification risk: combining public datasets can re-identify supposedly anonymized data. Famous examples include the Netflix Prize dataset and the AOL search logs.

Exam Tips: Answering Questions on Extracting Personal Data from Publicly Available Sources
1. Reject the 'it's public, so it's fine' trap.
The most common distractor states or implies that public data needs no protection. Under GDPR and most global regimes, this is wrong. Look for the answer that keeps privacy obligations in place.

2. Know the jurisdictional nuance.
If the question is clearly framed under US state law such as CCPA, publicly available information may genuinely be excluded. Check the definition carefully: lawfully made available government records, or data the consumer made public without restriction. Under GDPR or PIPEDA, assume protection applies.

3. Prefer the privacy-by-design, minimization-focused answer.
When asked what a technologist should do first or what is best, strong answers are usually:

• Conduct a PIA/DPIA
• Define the purpose
• Collect only the necessary fields
• Filter out sensitive data
• Pseudonymize early
• Document provenance

4. Watch for aggregation and secondary use.
Scenarios where separate public data points are combined into a profile point to aggregation harm. Scenarios where data is reused for a new purpose point to purpose limitation and secondary use.

5. Remember transparency for indirectly collected data.
Under GDPR, Article 14 requires informing data subjects when data was not collected from them. Answers claiming notice is never needed for scraped data are usually incorrect.

6. Spot special categories and biometrics.
Facial images, health signals, religion and political opinions raise the bar significantly. Expect references to Article 9, BIPA or Clearview AI-style scenarios. 'Manifestly made public' is narrow and does not cover data published by third parties.

7. Consider terms of service and technical controls.
An option that respects robots.txt, API terms and rate limits is generally more defensible than covert scraping that circumvents controls.

8. Think from both sides.
Some questions put you in the role of the platform. Choose layered anti-scraping controls, privacy-protective defaults and monitoring, not a single measure.

9. Link to the full data lifecycle.
The domain covers collection, use, dissemination and destruction. Good answers address retention limits, accuracy refreshes, controls on onward sharing and secure deletion. They should also include suppression lists to honor deletion requests.

10. Use elimination strategically.
Eliminate answers that are absolute, such as 'never allowed' or 'always permitted'. Also eliminate answers that rely solely on consent where consent is impractical, or that ignore risk assessment. The best answer typically balances business purpose with proportionate safeguards.

11. Recognize key cases and enforcement.

• Clearview AI: global fines and orders.
• hiQ v. LinkedIn: CFAA and scraping.
• Cambridge Analytica: data harvested through APIs, leading to a $5 billion FTC penalty for Facebook.
• The 2023 joint statement by global data protection authorities on scraping.

12. Keywords that signal the correct answer:
reasonable expectations, purpose limitation, data minimization, legitimate interests assessment, DPIA, transparency, provenance, de-identification, contextual integrity, aggregation risk.

Sample Question Walkthrough
A company plans to scrape public social media profiles of EU residents to build a marketing database. What should the privacy technologist recommend FIRST?

A) Proceed, because the data is publicly available
B) Conduct a DPIA and legitimate interests assessment, defining the purpose and the minimum data required
C) Obtain consent from each user after collection
D) Store the data in an encrypted database

Correct answer: B.

• A reflects the 'public equals free' fallacy.
• C is impractical, and after-the-fact consent cannot legitimize the collection.
• D is a valid security control but not the first step.

Summary
Extracting personal data from public sources is legal and ethical only when it is purposeful, proportionate, transparent and secured. Data minimization and privacy by design should be applied throughout the lifecycle. For the CIPT exam, assume protections apply unless a specific legal exclusion is clearly in play. Prioritize assessments and minimization, recognize aggregation and secondary-use harms, and choose answers that balance utility with respect for individuals' reasonable expectations.