Security and Privacy by Design for AI

Overview
Security and Privacy by Design for AI is a core concept within the ISACA Advanced in AI Risk (AAIR) domain of AI Life Cycle Risk Management. It describes the discipline of embedding security and privacy controls into every phase of an AI system's life cycle, from problem framing and data collection through model development, deployment, monitoring and retirement, rather than bolting them on after the system is built. For the exam, you must understand why this matters, what the principles are, how they are operationalized, and how to reason through scenario-based questions that test your judgment.

Why Security and Privacy by Design for AI Is Important
AI systems differ from conventional software in ways that create new and amplified risk:
1. Data dependency. Models are trained on large volumes of data, often including personal, sensitive or proprietary information. Poor data governance creates privacy exposure that is difficult to reverse once a model has learned from the data.
2. Novel attack surfaces. AI introduces threats that traditional application security does not address, such as data poisoning, model inversion, membership inference, model extraction (theft), adversarial examples, prompt injection and training data leakage.
3. Opacity and scale. Models can memorize and regurgitate training data, and because they operate at scale, a single flaw can affect millions of decisions or users.
4. Regulatory pressure. GDPR Article 25 (data protection by design and by default), the EU AI Act, sector regulations (HIPAA, GLBA, PCI DSS) and emerging AI laws explicitly require proactive, embedded controls. Retrofitting is costly and often legally insufficient.
5. Cost of late remediation. Fixing a privacy or security flaw after deployment may require retraining or decommissioning a model, destroying business value. Early integration is dramatically cheaper and more effective.
6. Trust and accountability. Stakeholders, regulators and customers expect demonstrable, auditable evidence that AI systems protect data and resist misuse. Design-stage controls produce that evidence.

What Security and Privacy by Design for AI Is
Security by Design means AI systems are architected so that confidentiality, integrity and availability are protected as default properties. It applies secure software development life cycle (SSDLC) principles to the AI pipeline and extends them to address model-specific threats.
Privacy by Design, originating with Ann Cavoukian's seven foundational principles and codified in GDPR Article 25 and ISO/IEC 31700, means personal data protection is proactive, the default setting, embedded into design, full-functionality (positive-sum), end-to-end across the life cycle, visible and transparent, and user-centric.
Applied to AI, these two disciplines converge into a single expectation: every AI life cycle decision should consider its security and privacy implications before it is made.

Key Principles You Must Know
- Proactive, not reactive: anticipate threats and privacy harms before they materialize.
- Privacy and security as the default: no user action required to be protected.
- Data minimization and purpose limitation: collect and retain only the data necessary for the defined purpose.
- Defense in depth: layered controls across data, model, infrastructure and application.
- Least privilege and zero trust: restrict access to data, model artifacts, training pipelines and inference endpoints.
- End-to-end life cycle protection: from data sourcing through decommissioning and secure deletion.
- Transparency and accountability: document decisions, maintain audit trails and enable oversight.
- Full functionality (positive-sum): security and privacy should not be traded off against business value; design for both.

How It Works: Integrating Controls Across the AI Life Cycle
1. Planning and problem definition
- Conduct a Privacy Impact Assessment (PIA) or Data Protection Impact Assessment (DPIA) and an AI-specific risk assessment.
- Define the lawful basis and purpose for data use; establish data classification requirements.
- Perform threat modeling using frameworks such as STRIDE, MITRE ATLAS (Adversarial Threat Landscape for AI Systems) and OWASP Top 10 for LLM Applications.
- Define security and privacy requirements as acceptance criteria, not afterthoughts.

2. Data collection and preparation
- Apply data minimization, consent management and purpose limitation.
- Use de-identification techniques: anonymization, pseudonymization, tokenization, masking and generalization.
- Apply Privacy-Enhancing Technologies (PETs): differential privacy (adding calibrated noise), synthetic data, federated learning (training without centralizing raw data), secure multi-party computation and homomorphic encryption.
- Validate data provenance and integrity to prevent data poisoning; maintain data lineage records.
- Encrypt data at rest and in transit; enforce role-based access control (RBAC) on training datasets.

3. Model development and training
- Secure the ML development environment (hardened notebooks, segregated environments, secrets management).
- Implement adversarial training and robustness testing to resist adversarial examples.
- Apply differentially private training (e.g., DP-SGD) where memorization risk is high.
- Scan third-party models, libraries and datasets for vulnerabilities; maintain an AI Bill of Materials (AIBOM) and manage AI supply chain risk.
- Version control models and data; sign model artifacts to guarantee integrity.

4. Validation and testing
- Perform security testing specific to AI: red teaming, adversarial testing, prompt injection testing, membership inference and model inversion tests, model extraction resistance.
- Test for training data leakage and memorization of personal data.
- Verify that privacy controls (e.g., data deletion requests, consent withdrawal) function end-to-end.
- Conduct bias and fairness testing, because discriminatory outputs often arise from the same poorly governed data that creates privacy risk.

5. Deployment
- Secure inference endpoints: authentication, authorization, rate limiting, input validation and output filtering to prevent prompt injection, jailbreaks and data exfiltration.
- Isolate models and apply API gateways; log inference requests in a privacy-respecting manner.
- Apply guardrails and content filters; restrict model access to downstream systems (least privilege for agents and plugins).
- Provide user-facing transparency (notices, explanations, opt-outs) consistent with privacy by default.

6. Monitoring and operations
- Continuously monitor for drift, anomalous queries, extraction attempts and data leakage.
- Integrate AI telemetry into the SIEM and incident response plan; define AI-specific incident playbooks.
- Re-run PIAs/DPIAs when purpose, data or model changes materially.
- Manage retraining pipelines securely to prevent poisoned feedback loops.

7. Retirement and decommissioning
- Securely delete or archive training data, model weights, embeddings and logs in line with retention schedules.
- Address machine unlearning or retraining obligations when individuals exercise erasure rights.
- Document decommissioning for audit purposes.

Supporting Frameworks and Standards
- NIST AI Risk Management Framework (AI RMF): Govern, Map, Measure, Manage, with Secure and Resilient and Privacy-Enhanced as trustworthiness characteristics.
- NIST Privacy Framework and NIST SP 800-53 privacy and security control families.
- ISO/IEC 42001 (AI management system), ISO/IEC 23894 (AI risk management), ISO/IEC 27001/27701 (information security and privacy information management), ISO/IEC 31700 (privacy by design for consumer products).
- GDPR Article 25, EU AI Act requirements for high-risk systems (data governance, robustness, cybersecurity, logging).
- OWASP Top 10 for LLM Applications and OWASP Machine Learning Security Top 10.
- MITRE ATLAS for adversarial tactics and techniques.
- ISACA COBIT and ISACA AI audit guidance for governance alignment.

Roles and Governance
Security and privacy by design requires collaboration among the data protection officer (DPO), CISO, AI risk owners, data scientists, ML engineers, legal, and internal audit. Governance artifacts include AI policies, model cards, data sheets, risk registers, PIAs, threat models and control test evidence. The three lines model applies: the first line builds and operates controls, the second line (risk, privacy, security) sets standards and oversees, and the third line (audit) provides independent assurance.

Common AI-Specific Threats and Matching Controls
- Data poisoning -> data provenance validation, anomaly detection on training data, access controls on pipelines.
- Model inversion / membership inference -> differential privacy, output perturbation, limiting confidence scores, query rate limiting.
- Model extraction (theft) -> API throttling, watermarking, monitoring query patterns, legal protections.
- Adversarial examples / evasion -> adversarial training, input preprocessing, ensemble methods, robustness testing.
- Prompt injection / jailbreaking -> input and output filtering, instruction hierarchy, least privilege for tool access, human-in-the-loop for sensitive actions.
- Training data leakage / memorization -> de-identification, DP training, output filtering, red teaming.
- Supply chain compromise -> model and dependency scanning, signed artifacts, AIBOM, trusted sources.
- Insecure plugins or agents -> sandboxing, scoped permissions, allow-lists.

Exam Tips: Answering Questions on Security and Privacy by Design for AI
1. Favor earliest-stage and preventive answers. When a question asks for the BEST or MOST effective way to address a security or privacy risk, the answer that embeds the control at the design or data-collection stage almost always beats detective or corrective controls applied after deployment. Proactive beats reactive is the core philosophy.

2. Look for data minimization first. If an option proposes collecting less data, reducing retention, or using de-identified or synthetic data, it is frequently the correct privacy answer because it eliminates risk rather than merely controlling it.

3. Distinguish privacy risks from security risks. Privacy concerns the appropriate use of personal data (purpose limitation, consent, individual rights, re-identification). Security concerns protecting assets from unauthorized access or manipulation (confidentiality, integrity, availability). A question about model extraction is a security question; a question about re-identifying individuals from model outputs is primarily a privacy question (though controls overlap). Match the control to the risk type.

4. Know AI-specific threats and their canonical controls. Expect questions that name a threat (poisoning, inversion, prompt injection) and ask for the most appropriate mitigation. Memorize the threat-to-control pairings above. Avoid generic answers like install antivirus when an AI-specific control is offered.

5. Recognize PETs and when they apply. Differential privacy protects against inference from outputs or models; federated learning avoids centralizing raw data; homomorphic encryption allows computation on encrypted data; synthetic data reduces exposure in development and testing. If the scenario emphasizes that data cannot leave jurisdictions or organizations, think federated learning. If it emphasizes preventing individual records from being inferred, think differential privacy.

6. Remember the governance angle. ISACA questions often test the risk practitioner's role: who should own a decision, what artifact provides evidence, and when a PIA/DPIA should be performed or updated (answer: before processing begins, and again when purpose, data or technology changes materially). The risk practitioner advises and assures; business owners accept risk.

7. Life cycle framing. If a question describes a control being added at a given phase, ask whether an earlier phase would have been more appropriate. Conversely, if a model is already deployed, the best immediate action may be a compensating control (rate limiting, output filtering, enhanced monitoring) while a redesign is planned. Read carefully for the words FIRST, NEXT, BEST, MOST and PRIMARY.

8. Apply the full-functionality principle. Options that simply block the AI initiative or remove functionality are rarely best. Prefer answers that achieve both business value and protection.

9. Tie controls to regulatory drivers. If a scenario mentions EU residents or personal data, GDPR Article 25 and DPIA requirements apply. If it mentions high-risk AI use cases (employment, credit, biometrics), think EU AI Act obligations for data governance, logging, robustness and cybersecurity.

10. Eliminate distractors. Wrong answers typically (a) address a different risk than the one described, (b) occur too late in the life cycle, (c) rely solely on policy or training without technical enforcement, or (d) are generic IT controls that ignore AI-specific threats. Use these patterns to narrow choices.

Sample Question Walkthrough
An organization is developing a model that will be trained on patient records. Which of the following is the BEST way to reduce the risk that the deployed model discloses information about individual patients?
A. Encrypt the model weights at rest.
B. Apply differential privacy during training and de-identify the training data.
C. Require users to accept terms of use before querying the model.
D. Monitor inference logs for suspicious queries after deployment.

Analysis: The risk described is disclosure of individual information through model outputs (membership inference or memorization), a privacy risk. Option A protects stored weights but not outputs. Option C is a policy control with no technical enforcement. Option D is detective and post-deployment. Option B embeds controls at the data and training stage and directly addresses inference risk. Correct answer: B.

Key Takeaways
- Security and privacy by design means controls are embedded proactively across the entire AI life cycle, not added later.
- AI introduces unique threats (poisoning, inversion, extraction, adversarial inputs, prompt injection, leakage) that require AI-specific controls alongside traditional security.
- Data minimization, de-identification and PETs are first-line privacy controls; threat modeling, secure pipelines, adversarial testing and endpoint hardening are first-line security controls.
- PIAs/DPIAs, threat models, model documentation and audit trails provide the governance evidence regulators and auditors expect.
- On the exam, choose the earliest, most preventive, AI-specific and technically enforceable option that preserves business functionality and matches the specific risk described.