Privacy Risks in AI and Machine Learning

Privacy Risks in AI and Machine Learning (CDPSE - Privacy Engineering)

Overview
Privacy Risks in AI and Machine Learning is a key topic in the Privacy Engineering domain of the ISACA Certified Data Privacy Solutions Engineer (CDPSE) exam. It covers how AI/ML systems collect, process, infer, and expose personal data. It also covers the technical and organizational controls a privacy engineer uses to reduce those risks across the AI lifecycle.

Why It Is Important
AI and ML systems are built on data, and very often that data is personal. Unlike traditional systems, ML models can:
- Memorize training data and later leak it.
- Infer sensitive attributes (health, sexual orientation, political views) that were never collected directly.
- Make automated decisions with legal or similarly significant effects on individuals (credit, hiring, insurance).
- Repurpose data beyond the purpose for which it was collected.
- Operate as black boxes, which makes transparency and accountability hard.

Regulations increasingly target these risks. Examples include:
- GDPR Article 22 (automated decision-making and profiling)
- GDPR Articles 13-15 (meaningful information about the logic involved)
- The EU AI Act (risk-based classification of AI systems)
- CCPA/CPRA rules on automated decision-making
- The NIST AI Risk Management Framework

Organizations face regulatory fines, reputational damage, discrimination claims, and loss of trust if AI privacy risks are not managed. CDPSE candidates are expected to embed privacy into AI solutions by design, not bolt it on afterward.

What It Is
Privacy risks in AI/ML are the potential harms to individuals arising from how personal data is used across the ML lifecycle:
1. Data collection and sourcing
2. Data preparation and labeling
3. Model training
4. Model validation and testing
5. Deployment and inference
6. Monitoring, retraining, and decommissioning

Key Privacy Risks

1. Excessive data collection (violating data minimization)
Teams often gather as much data as possible on the assumption that more data means better models. This conflicts with minimization and purpose limitation.

2. Purpose creep / secondary use
Data collected for one purpose (e.g., customer service) is reused to train models for another (e.g., marketing profiling) without a lawful basis or notice.

3. Lack of valid consent or lawful basis
Web-scraped data or data from third-party brokers may lack proper consent or provenance.

4. Re-identification
Anonymized or pseudonymized datasets can be re-identified by combining them with other datasets. ML is very good at the pattern-matching that makes this possible.

5. Inference and profiling
Models can derive sensitive information, such as predicting pregnancy from shopping habits. Inferred data is still personal data.

6. Model memorization and data leakage
Large models, especially generative AI and LLMs, can reproduce verbatim training data, such as names, addresses, or credentials.

7. Privacy attacks on models
- Membership inference attack: determines whether a specific individual's record was in the training set.
- Model inversion attack: reconstructs sensitive features or representative inputs (e.g., faces) from model outputs.
- Attribute inference: infers hidden attributes of individuals.
- Model extraction/stealing: replicates the model by querying it, which can enable further attacks.
- Data poisoning: corrupts training data to manipulate outcomes. This is mainly an integrity issue, but it has privacy consequences.
- Prompt injection (generative AI): tricks a model into revealing confidential data or system prompts.

8. Bias and discrimination
Biased training data produces unfair outcomes for protected groups. This links privacy to fairness and ethics.

9. Lack of transparency and explainability
Individuals may have a right to meaningful information about automated decisions. Opaque models make this difficult.

10. Difficulty honoring data subject rights
Erasure (right to be forgotten), rectification, and objection are hard once data is embedded in model weights. Machine unlearning and retraining are emerging responses.

11. Third-party and supply chain risk
Pre-trained models, AI APIs, and SaaS AI tools may send data to vendors, use it for their own training, or store it in other jurisdictions.

12. Shadow AI
Employees may paste personal or confidential data into public generative AI tools without approval.

13. Retention
Training datasets, logs, prompts, and outputs are often kept indefinitely.

How It Works: Managing Privacy Risks in AI/ML

Step 1: Governance and accountability
- Establish an AI governance policy, an acceptable use policy for AI, and clear roles (data owner, model owner, DPO/privacy officer).
- Maintain an inventory of AI systems and their data flows.
- Align with frameworks such as the NIST AI RMF (Govern, Map, Measure, Manage), ISO/IEC 42001 (AI management systems), ISO/IEC 23894 (AI risk management), and the OECD AI Principles.

Step 2: Privacy Impact Assessment (PIA) / DPIA
- Perform a DPIA before developing or deploying AI that processes personal data, especially for profiling, large-scale processing, sensitive data, or automated decisions.
- Identify the lawful basis, purpose, data categories, risks, and mitigations.
- Consider an Algorithmic Impact Assessment that also covers fairness.

Step 3: Data-level controls (privacy by design)
- Data minimization: use only the features necessary for the model's purpose.
- De-identification: anonymization, pseudonymization, tokenization, masking, generalization (k-anonymity, l-diversity, t-closeness).
- Synthetic data: generate artificial data that keeps statistical properties without real individuals. Note that it can still leak if poorly generated.
- Data provenance and lineage: document the source, consent, and lawful basis for training data.
- Labeling and classification of sensitive data before ingestion.

Step 4: Privacy-Enhancing Technologies (PETs) for ML
- Differential privacy: adds calibrated statistical noise during training (e.g., DP-SGD) or to outputs. It mathematically limits what can be learned about any one individual and is the strongest defense against membership inference.
- Federated learning: trains models across decentralized devices or sites, so raw data never leaves its source; only model updates are shared. It is often combined with secure aggregation and differential privacy, because updates can still leak information.
- Homomorphic encryption: allows computation on encrypted data.
- Secure multi-party computation (SMPC): lets multiple parties jointly compute without revealing their inputs.
- Trusted execution environments (confidential computing): protect data in use.

Step 5: Model-level controls
- Regularization and early stopping to reduce overfitting and memorization.
- Limit output detail, e.g., return labels instead of full confidence scores, to reduce inversion and membership inference.
- Rate limiting and query monitoring to deter model extraction.
- Red-teaming and privacy testing (simulated membership inference, prompt-injection testing).
- Output filtering and guardrails for generative AI to block PII disclosure.

Step 6: Transparency, explainability, and human oversight
- Provide clear privacy notices about AI use and the logic involved.
- Use explainability techniques (e.g., SHAP, LIME) and document models with model cards and datasheets for datasets.
- Implement human-in-the-loop review for significant automated decisions.
- Give individuals the ability to contest decisions.

Step 7: Data subject rights
- Design processes to locate personal data in training sets.
- Plan for retraining or machine unlearning when erasure requests affect models.
- Honor opt-outs from profiling.

Step 8: Third-party management
- Perform vendor due diligence on AI providers.
- Use contract clauses that prohibit the vendor from using your data to train its models.
- Address data residency, sub-processors, and audit rights.

Step 9: Monitoring and lifecycle management
- Monitor for model drift, bias, and unexpected inferences.
- Apply retention schedules to training data, logs, and prompts.
- Securely decommission models and associated data.

How to Answer Exam Questions on This Topic
CDPSE questions are usually scenario-based. They ask what the privacy engineer should do FIRST, what is the BEST control, or what is the GREATEST risk. Use this approach:
1. Identify the lifecycle stage in the scenario: collection, training, deployment, or monitoring.
2. Identify the specific risk: re-identification, inference, memorization, secondary use, lack of transparency, or third-party exposure.
3. Match the control to the risk.
- Membership inference → differential privacy.
- Raw data cannot leave hospitals or devices → federated learning.
- Need realistic test data → synthetic data.
- Automated decisions with significant effects → human review and the right to contest.
- Black-box model → explainability and documentation.
4. Prefer preventive, design-stage controls over detective or after-the-fact ones.
5. Remember the governance sequence: understand the data, purpose, and lawful basis, and assess risk (PIA/DPIA), before choosing technical controls.

Exam Tips: Answering Questions on Privacy Risks in AI and Machine Learning
- The DPIA/PIA comes first. When a scenario describes a new AI initiative using personal data, the best first step is usually to conduct a privacy impact assessment. Do not jump straight to selecting technology.
- Data minimization is a core principle. If an answer reduces the personal data used while still meeting the business purpose, it is often correct.
- Purpose limitation catches secondary use. If existing data is being reused to train a model for a new purpose, look for answers about verifying compatibility, lawful basis, consent, or updated notice.
- Differential privacy is the classic answer for protecting individuals in aggregate outputs and preventing membership inference.
- Federated learning keeps data local, but it is not a complete solution on its own. Model updates can leak, so secure aggregation or differential privacy may be needed.
- Anonymization is not foolproof. Expect questions highlighting re-identification risk through linkage. Pseudonymized data is still personal data under GDPR.
- Inferred data is personal data. Sensitive inferences must be protected like collected sensitive data.
- Automated decision-making requires safeguards: human intervention, the right to contest, and meaningful explanation (GDPR Art. 22).
- Explainability supports transparency and accountability. Choose answers that document models and inform data subjects.
- Generative AI questions often focus on prompt data leakage, vendors training on customer data, and output disclosure of PII. Best answers include acceptable use policies, approved enterprise tools, contractual restrictions, and input/output filtering.
- Third-party AI risk means vendor assessment, contracts, data processing agreements, and audit rights.
- Bias is related but distinct. If a question emphasizes discriminatory outcomes, look for training-data review, fairness testing, and diverse datasets. Privacy and ethics often overlap in these answers.
- Distinguish attack types. Membership inference asks whether a person was in the data. Model inversion reconstructs the data. Model extraction copies the model. Poisoning corrupts training data.
- Watch qualifier words such as FIRST, BEST, MOST, and GREATEST. Several answers may be valid controls; pick the one that addresses root cause at the earliest stage with the broadest effect.
- Avoid extreme answers. Answers like banning all AI or deleting the entire model are rarely best unless the scenario describes an unlawful situation. Risk-based, proportionate controls are preferred.
- Think lifecycle. Privacy must be addressed from design through decommissioning, including retention of training data, logs, and prompts.
- Accountability is organizational. Senior management and data owners are accountable; the privacy engineer implements and advises.

Quick Review Summary
AI/ML amplifies privacy risks through large-scale data use, inference, memorization, opacity, and automated decisions. Manage them through:
- Governance and DPIAs
- Minimization and purpose limitation
- De-identification and synthetic data
- PETs (differential privacy, federated learning, homomorphic encryption, SMPC)
- Model hardening
- Transparency and human oversight
- Support for data subject rights
- Vendor controls
- Continuous monitoring

On the exam, identify the risk, map it to the most appropriate preventive control, and respect the governance-first sequence.