AI Threat Modeling

1. Why AI Threat Modeling Matters

Artificial intelligence systems introduce an attack surface that traditional security and risk programs were never designed to cover. A conventional application has code, infrastructure and data; an AI system adds training data, learned model parameters, inference pipelines, prompts, outputs that influence decisions, and often autonomous actions. Each of these can be manipulated by an adversary in ways that do not look like classic hacking. A model can be poisoned so it misclassifies a specific face, a large language model can be tricked into leaking confidential data through a crafted prompt, or a competitor can reconstruct a proprietary model simply by querying an API thousands of times.

Within the ISACA AAIR body of knowledge, AI threat modeling sits inside the AI Risk Program Management domain because it is the structured discipline that converts vague concern about 'AI security' into a documented, repeatable, risk-ranked list of threats and controls. Without it, an AI risk program cannot credibly claim to know what it is protecting, from whom, and whether its controls are proportionate. Regulators (EU AI Act, NIST AI RMF, ISO/IEC 42001 and 23894) expect organizations to demonstrate that AI-specific threats were identified and treated. Threat modeling is the evidence.

Key reasons it is important:
• Novel threat categories – adversarial, data, and model-centric attacks are not caught by network scanners or code review.
• Early and cheap – threats found at design time cost far less to mitigate than those found after deployment or after a public incident.
• Prioritization – AI teams cannot mitigate everything; threat modeling ranks what matters.
• Shared language – it brings data scientists, security engineers, risk managers, legal and the business onto one view of the system.
• Assurance and accountability – it produces artifacts auditors and regulators can examine, supporting the governance expectations in AAIR.

2. What AI Threat Modeling Is

Definition: AI threat modeling is a structured, systematic process for identifying, analyzing, prioritizing and documenting the ways an AI system can be attacked, misused or made to fail, together with the controls that reduce those risks to an acceptable level. It is performed throughout the AI lifecycle, not once.

It answers four classic questions, adapted to AI:
1. What are we building? – Decompose the AI system: data sources, pipelines, features, model type, training environment, inference endpoints, prompts, plug-ins/tools, downstream consumers, human oversight points.
2. What can go wrong? – Enumerate threats using AI-aware frameworks.
3. What are we going to do about it? – Select and assign mitigations.
4. Did we do a good job? – Validate, test (including red teaming) and revisit as the system changes.

How it differs from traditional threat modeling:
• Assets include training data, labels, model weights, hyperparameters, embeddings and prompts – not just servers and credentials.
• Trust boundaries include the boundary between the model and any untrusted input (user prompts, web content, retrieved documents) because the model itself can be socially engineered.
• Failure modes include statistical and behavioral harms (bias, drift, hallucination, unsafe outputs) that are not strictly 'attacks' but are often included in scope because they can be exploited or cause harm.
• Adversaries may need no system access at all – they interact only through the public inference interface.

3. Core AI Threat Categories You Must Know

Organized roughly by lifecycle stage:

Data and training stage
• Data poisoning – injecting or altering training data to degrade accuracy or implant a hidden behavior.
• Backdoor / Trojan attacks – a specific trigger (pattern, token, pixel patch) causes attacker-chosen output while normal behavior appears intact.
• Label flipping – corrupting labels in supervised learning.
• Supply chain compromise – malicious or vulnerable pre-trained models, datasets, libraries, model hubs, or fine-tuning services; serialized model files (e.g., pickle) that execute code when loaded.

Model stage
• Model extraction / stealing – reconstructing a functionally equivalent model through repeated queries.
• Model inversion – recovering sensitive features of training data from model outputs.
• Membership inference – determining whether a specific record was in the training set (a privacy breach).
• Model theft – direct exfiltration of weights.

Inference / deployment stage
• Evasion / adversarial examples – minimally perturbed inputs that cause misclassification (e.g., altered stop signs, malware repacking).
• Prompt injection (direct and indirect) – instructions embedded in user input or in retrieved/third-party content that override system instructions in LLMs.
• Jailbreaking – bypassing safety alignment and content guardrails.
• Sensitive information disclosure – leakage of system prompts, training data, or data from other users.
• Insecure output handling – downstream systems blindly trusting model output, enabling XSS, SQL injection or command execution.
• Excessive agency – agents or plug-ins with more permissions or autonomy than needed, turning a manipulated output into a real-world action.
• Denial of service / resource exhaustion – expensive or recursive prompts, sponge examples that spike compute cost.

Cross-cutting
• Misuse – legitimate capability used for harm (disinformation, fraud, bioweapon uplift).
• Over-reliance – humans accepting hallucinated or biased outputs without verification.
• Drift and degradation – natural change in data over time that an attacker can also accelerate.
• Insider threat – data scientists or MLOps staff altering data, code or weights.

4. Frameworks and Reference Taxonomies

The exam expects recognition of the major sources used to populate a threat model:
• MITRE ATLAS (Adversarial Threat Landscape for AI Systems) – an ATT&CK-style matrix of adversary tactics and techniques specific to ML, with real-world case studies. Best for mapping how an attacker proceeds.
• OWASP Top 10 for LLM Applications – prompt injection, insecure output handling, training data poisoning, model DoS, supply chain, sensitive information disclosure, insecure plug-in design, excessive agency, over-reliance, model theft.
• OWASP Machine Learning Security Top 10 – input manipulation, data poisoning, model inversion, membership inference, model theft, supply chain, transfer learning attack, model skewing, output integrity, model poisoning.
• STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) – the classic Microsoft model, adapted so that 'tampering' covers data/model poisoning and 'information disclosure' covers inversion and membership inference.
• NIST AI 100-2 (Adversarial Machine Learning taxonomy) – formal classification of attacks by attacker goals (availability, integrity, privacy, abuse) and capabilities.
• NIST AI RMF (Govern, Map, Measure, Manage) – threat modeling supports the Map and Measure functions.
• ISO/IEC 23894 and ISO/IEC 42001 – position threat and risk identification within an AI risk management process and management system.
• PASTA, LINDDUN – risk-centric and privacy-centric methods sometimes layered on for impact analysis and privacy threats.

5. How AI Threat Modeling Works – Step by Step

Step 1 – Define scope and objectives. Decide which system, which version, which use cases, and which harms are in scope (security only, or also safety, privacy, fairness). Align with the organization's AI risk appetite and risk taxonomy defined by the AI risk program.

Step 2 – Assemble the team. Threat modeling is cross-functional: product owner, data scientist/ML engineer, security architect, privacy officer, risk manager, and where relevant legal/compliance and domain experts. The AI risk function typically facilitates and owns the methodology; system owners own the results.

Step 3 – Decompose the system. Create data flow diagrams showing: data sources and their trust level; ingestion and preprocessing; feature stores; training pipeline and environment; model registry; deployment/inference endpoints; retrieval sources (RAG); tools, APIs and agents the model can call; human-in-the-loop points; logging and monitoring; and consumers of outputs. Mark trust boundaries explicitly, especially where untrusted content reaches the model.

Step 4 – Identify assets and adversaries. Assets: training data, PII, labels, weights, prompts, intellectual property, output integrity, availability, user trust. Adversaries: external attackers, competitors, malicious users, insiders, compromised suppliers, and even non-malicious causes (drift, bugs). Capture their goals (availability, integrity, confidentiality/privacy, abuse) and capabilities (black-box query only, white-box, data access, pipeline access).

Step 5 – Enumerate threats. Walk each component and boundary against a taxonomy (ATLAS, OWASP, STRIDE, NIST AML). For each, write a threat statement: 'An [adversary] with [access] could [action] causing [impact].' Include misuse and abuse cases, not just technical exploits.

Step 6 – Analyze and prioritize. Rate likelihood (attacker motivation, skill required, exposure of the interface, existing controls) and impact (safety, financial, legal, privacy, reputational, scale of affected users). Use the organization's risk matrix or a scoring scheme (DREAD, CVSS-like adaptations, or qualitative High/Medium/Low). Map to inherent and residual risk so the program can track treatment.

Step 7 – Select mitigations. Typical AI-specific controls:
• Data provenance, integrity checks, anomaly detection on training data, dataset versioning and signing.
• Adversarial training, input validation and sanitization, ensemble methods, randomized smoothing.
• Rate limiting, query monitoring and output perturbation against extraction and inference attacks.
• Differential privacy, data minimization, synthetic data for privacy threats.
• Prompt hardening, instruction hierarchy, content filtering, allow-listed tools, output encoding and validation, least-privilege agent permissions, human approval for consequential actions.
• Model and artifact signing, SBOM/AIBOM, safe serialization formats, vetted model sources.
• Access control and secrets management for registries and pipelines; separation of duties for data scientists and MLOps.
• Monitoring for drift, anomalous inputs, and abuse; incident response playbooks for AI incidents.
Assign an owner, due date and acceptance criteria for each mitigation; document risk acceptance where mitigation is not feasible.

Step 8 – Validate. Confirm controls work through security testing, AI red teaming, adversarial robustness evaluations, privacy attack simulations and penetration testing of the surrounding application. Red teaming is the verification of the threat model, not a substitute for it.

Step 9 – Document and integrate. Store the threat model with the model card / system card, risk register and change management records. Feed findings into the AI risk register, control library, training, and third-party risk assessments.

Step 10 – Iterate. Re-run threat modeling on triggers: new model version, retraining, new data source, new plug-in or tool, change of deployment context, new user population, emerging attack technique, or a security incident. Continuous threat modeling is a program expectation, not a one-off deliverable.

6. Where It Fits in an AI Risk Program

• Governance: policy mandates threat modeling for AI systems above a defined risk tier; roles and RACI are set.
• Risk identification: threat modeling is a primary input to the AI risk register.
• Risk assessment: provides likelihood and impact rationale.
• Risk response: yields control requirements and secure design decisions.
• Monitoring and reporting: metrics such as percentage of in-scope AI systems with a current threat model, open high-rated threats, mean time to mitigate.
• Third-party and supply chain: extends to vendor models and APIs; contractual rights to assurance evidence.
• Lifecycle integration: embedded into MLOps/secure development gates (design review, pre-deployment release, post-deployment review).

7. Common Pitfalls
• Treating AI threat modeling as a purely technical security task and excluding data scientists or the business.
• Reusing a web-application threat model unchanged, missing data, model and prompt attack vectors.
• Modeling only the model, not the full system (retrieval sources, tools, downstream automation).
• Ignoring indirect prompt injection from retrieved or third-party content.
• Doing it once at launch and never after retraining.
• Producing a threat list with no owners, ratings or mitigations – no linkage to the risk register.
• Confusing red teaming (testing) with threat modeling (analysis and design).

8. Exam Tips: Answering Questions on AI Threat Modeling

Tip 1 – Know the lifecycle mapping. Many questions describe a scenario and ask for the most likely threat. Anchor on the stage: training data manipulated = poisoning/backdoor; crafted input at inference = evasion/adversarial example or prompt injection; repeated API queries to copy behavior = model extraction; determining if a person's record was used = membership inference; recovering faces or text from outputs = model inversion.

Tip 2 – Distinguish look-alike terms.
• Poisoning (training-time, integrity of the model) vs. evasion (inference-time, integrity of a prediction).
• Model inversion (reconstructs data attributes) vs. membership inference (confirms presence of a record) vs. model extraction (copies the model).
• Direct prompt injection (from the user) vs. indirect (from content the model reads) vs. jailbreak (defeating safety alignment).
• Threat modeling (structured analysis, design-time, proactive) vs. red teaming (adversarial testing, validation) vs. penetration testing (infrastructure/application focused).

Tip 3 – 'When' questions favor early and continuous. If asked when threat modeling should occur, the best answer is during design, and repeated at significant change – never 'after deployment only' or 'only when an incident occurs'.

Tip 4 – 'Who' questions favor cross-functional with clear ownership. The AI risk or security function typically facilitates and sets methodology; the system or product owner is accountable for the risk; data scientists provide technical input. Answers that assign it solely to one silo are usually wrong.

Tip 5 – 'First step' questions. The first step is almost always to define scope and understand/decompose the system (identify assets, data flows, trust boundaries) – not to select tools, buy controls, or start testing.

Tip 6 – 'Best / most effective control' questions. Match the control to the threat precisely:
• Poisoning → data provenance, validation, integrity checks.
• Extraction → rate limiting, query monitoring, output perturbation.
• Membership inference / inversion → differential privacy, limiting confidence scores.
• Evasion → adversarial training, input preprocessing.
• Prompt injection → input/output filtering, instruction separation, least-privilege tools, human approval for actions.
• Supply chain → vetted sources, signed artifacts, safe serialization, AIBOM.
Prefer preventive, design-level controls over detective ones unless the question specifies detection.

Tip 7 – Framework recognition. ATLAS = adversary tactics/techniques for ML (ATT&CK style). OWASP LLM Top 10 = application-level LLM risks. STRIDE = generic categories adapted to AI. NIST AI RMF = Map/Measure functions. If a question asks for a framework to enumerate adversary techniques against ML, choose MITRE ATLAS; for LLM application vulnerabilities, choose OWASP Top 10 for LLM.

Tip 8 – Think like a risk manager, not just an engineer. ISACA exam answers favor: alignment with risk appetite, documentation in the risk register, assigned ownership, residual risk acceptance by the right authority, and integration with governance. Between a technically clever answer and a governance-complete answer, the governance-complete one usually wins.

Tip 9 – Watch for scope traps. If a scenario mentions RAG, plug-ins, agents or third-party models, the correct answer typically expands the threat model to those components (indirect injection, excessive agency, supply chain) rather than focusing on the core model alone.

Tip 10 – Eliminate absolutes. Options containing 'eliminates all risk', 'only', 'never needs updating', or 'replaces the need for' are rarely correct. Threat modeling reduces and prioritizes risk; it does not remove it.

Tip 11 – Read for the stem's keyword. 'Primary', 'most important', 'first', 'best' change the correct answer. 'Primary purpose' of AI threat modeling = systematic identification and prioritization of threats to inform controls. 'Primary benefit' = early, cost-effective identification of AI-specific risks and defensible evidence for stakeholders.

Tip 12 – Remember the output. The deliverables of threat modeling are: system/data flow diagrams with trust boundaries, an asset and adversary inventory, a prioritized threat list with risk ratings, mapped mitigations with owners, and residual risk documentation feeding the AI risk register. A question asking what a threat model should produce is looking for this set, not just 'a list of vulnerabilities'.

9. Quick Revision Summary
• AI threat modeling = structured, lifecycle-wide identification, analysis, prioritization and treatment of AI-specific threats.
• Unique AI assets: data, labels, weights, prompts, outputs, autonomy.
• Key threats: poisoning, backdoors, evasion, extraction, inversion, membership inference, prompt injection, insecure output handling, excessive agency, supply chain, misuse, over-reliance.
• Key frameworks: MITRE ATLAS, OWASP LLM Top 10, OWASP ML Top 10, STRIDE, NIST AI 100-2, NIST AI RMF, ISO/IEC 23894/42001.
• Process: scope → team → decompose → assets/adversaries → enumerate → prioritize → mitigate → validate (red team) → document → iterate.
• Program linkage: policy-mandated, risk-tiered, feeds the AI risk register, tracked by metrics, repeated on change.
• Exam mindset: earliest, most systematic, cross-functional, control-matched, governance-complete answer is usually correct.