Threat Modeling for AI Systems

Overview
Threat modeling for AI systems is a core topic within the AI Risk Management for Security domain of the ISACA Advanced in AI Security Management (AAISM) certification. It represents the structured, proactive practice of identifying what can go wrong with an AI system, who might cause it, how they could do it, and what should be done about it before an incident occurs. For the exam, you must understand both traditional threat modeling concepts and the unique ways they must be adapted for machine learning and generative AI.

Why Threat Modeling for AI Systems Is Important
AI systems introduce an attack surface that classical application security does not fully address. A conventional web application can be hardened by validating input, patching libraries and controlling access. An AI system, by contrast, learns its behavior from data, makes probabilistic rather than deterministic decisions, and can be manipulated through the data it is trained on, the inputs it receives at inference time, and the outputs it produces. Key reasons threat modeling matters for AI include:
• Novel attack vectors: Data poisoning, adversarial examples, model inversion, membership inference, model extraction and prompt injection have no exact equivalent in traditional software.
• Opacity: Many models are black boxes, so vulnerabilities cannot be found simply by reading code; they must be reasoned about systematically.
• Supply chain exposure: Pre-trained models, third-party datasets, open-source frameworks and external APIs expand the trust boundary far beyond the organization.
• Regulatory and governance pressure: Frameworks such as the NIST AI Risk Management Framework (AI RMF), ISO/IEC 42001, ISO/IEC 23894 and the EU AI Act expect organizations to identify and manage AI risk in a documented, repeatable way. Threat modeling provides the evidence trail.
• Cost efficiency: Fixing a design flaw during the design phase is dramatically cheaper than retraining a compromised model or responding to a breach of sensitive training data.
• Trust and safety: Harmful outputs, biased decisions and privacy leakage erode stakeholder trust and can cause legal, reputational and physical harm.

What Threat Modeling for AI Systems Is
Threat modeling is a structured process for analyzing a system from an adversary's perspective to identify threats, vulnerabilities and countermeasures. ISACA generally frames it around four guiding questions:
1. What are we building? (Decompose and understand the system.)
2. What can go wrong? (Identify threats.)
3. What are we going to do about it? (Define mitigations and controls.)
4. Did we do a good enough job? (Validate and iterate.)

When applied to AI, the scope expands to cover the full AI lifecycle: problem definition, data collection and labeling, feature engineering, model training and tuning, validation, deployment, inference, monitoring, retraining and decommissioning. Every stage introduces assets to protect and threats to consider. The key assets in an AI system typically include training data, labels, feature pipelines, model weights and architecture, hyperparameters, prompts and system instructions, inference APIs, output logs, vector databases, and the downstream decisions or actions the model influences.

AI-Specific Threat Categories You Must Know
• Data poisoning: An attacker corrupts training or fine-tuning data so the model learns incorrect or malicious behavior. Includes backdoor or trojan attacks that trigger on a specific pattern.
• Evasion / adversarial examples: Carefully perturbed inputs at inference time cause misclassification (e.g., a modified stop sign image read as a speed limit sign).
• Model inversion: An attacker reconstructs sensitive training data from model outputs.
• Membership inference: An attacker determines whether a specific record was in the training set, creating a privacy breach.
• Model extraction / theft: Repeated queries are used to replicate a proprietary model, stealing intellectual property.
• Prompt injection (direct and indirect): Malicious instructions embedded in user input or in retrieved content cause an LLM to ignore its system prompt, leak data or take unauthorized actions.
• Insecure output handling: Model outputs are passed to downstream systems without validation, enabling injection or code execution.
• Sensitive information disclosure: Models reveal confidential training data, system prompts or PII.
• Supply chain compromise: Tampered pre-trained models, poisoned public datasets, malicious plugins or vulnerable ML libraries.
• Excessive agency: AI agents granted too many permissions or tools take harmful autonomous actions.
• Denial of service / resource exhaustion: Crafted inputs consume excessive compute or exploit long context windows.
• Model drift and degradation: Not always adversarial, but a reliability threat that must be monitored.

Frameworks and Reference Sources
• STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege): The classic Microsoft model, readily mapped to AI (e.g., Tampering = data poisoning; Information Disclosure = model inversion).
• MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems): A knowledge base of adversary tactics, techniques and real-world case studies specific to AI, modeled on MITRE ATT&CK.
• OWASP Top 10 for LLM Applications and OWASP Machine Learning Security Top 10: Prioritized lists of the most critical AI and LLM vulnerabilities.
• NIST AI RMF (Govern, Map, Measure, Manage) and NIST AI 100-2 (Adversarial Machine Learning taxonomy): Provide risk management structure and standardized terminology.
• PASTA (Process for Attack Simulation and Threat Analysis): A risk-centric, seven-stage methodology aligning business objectives with technical threats.
• LINDDUN: A privacy-focused threat modeling framework highly relevant to AI data handling.
• Attack trees: Hierarchical diagrams that decompose an attacker goal into required sub-steps.
• ISO/IEC 23894 and ISO/IEC 42001: International guidance for AI risk management and AI management systems.

How Threat Modeling for AI Systems Works: Step by Step
Step 1 - Define scope and objectives. Clarify the business purpose of the AI system, its criticality, regulatory context, and the risk appetite of the organization. Identify stakeholders: data scientists, ML engineers, security, privacy, legal, compliance and business owners.

Step 2 - Decompose the system. Create data flow diagrams (DFDs) or architecture diagrams that show data sources, ingestion pipelines, feature stores, training environments, model registries, deployment endpoints, inference APIs, plugins or tools, retrieval components (e.g., RAG vector stores), human-in-the-loop points and downstream consumers. Mark trust boundaries, especially where third-party data, pre-trained models or external users enter the system.

Step 3 - Identify and classify assets. Determine what has value and what must be protected: training data confidentiality and integrity, model intellectual property, system prompt secrecy, availability of the inference service, integrity of decisions and safety of outputs.

Step 4 - Identify threats and threat actors. Apply a framework such as STRIDE across each component and trust boundary, then enrich with AI-specific techniques from MITRE ATLAS and OWASP. Consider threat actors ranging from external attackers and competitors to malicious insiders, careless users and compromised suppliers. Use attack trees or misuse cases to describe attack paths.

Step 5 - Assess and prioritize risk. Evaluate each threat by likelihood and impact. Consider attacker capability, required access (white-box vs. black-box), ease of exploitation, detectability and consequences including safety, privacy, financial, legal and reputational harm. Scoring approaches such as DREAD, CVSS-style ratings or qualitative heat maps are acceptable; the key is consistency and alignment with enterprise risk management.

Step 6 - Define mitigations and controls. Select countermeasures mapped to the threats, for example:
• Data poisoning: data provenance tracking, validation and sanitization, anomaly detection on training data, access controls on data pipelines, dataset signing.
• Adversarial examples: adversarial training, input preprocessing, ensemble models, confidence thresholds and human review.
• Model inversion / membership inference: differential privacy, output perturbation, limiting confidence score exposure, data minimization.
• Model extraction: rate limiting, query monitoring, watermarking, API authentication and anomaly detection.
• Prompt injection: input and output filtering, privilege separation between instructions and data, least-privilege tool access, content moderation layers, human approval for high-impact actions.
• Supply chain: model and dataset provenance (model cards, datasheets, SBOM/AI-BOM), integrity verification, trusted repositories and vulnerability scanning.
• General: logging and monitoring of inference behavior, drift detection, red teaming, incident response playbooks for AI.

Step 7 - Validate and iterate. Confirm that controls were implemented and effective through testing, AI red teaming, penetration testing and metrics. Threat models are living documents: revisit them when data sources change, models are retrained, new features or plugins are added, or new attack techniques emerge.

Step 8 - Document and communicate. Record threats, decisions, residual risks and ownership. Feed results into the risk register, governance committees and compliance reporting. Residual risk must be formally accepted by an appropriate risk owner.

Integration With AI Governance and the Lifecycle
Threat modeling should be embedded in the secure development lifecycle (shift-left) and in MLOps/LLMOps pipelines rather than treated as a one-time gate. It supports the Map and Measure functions of the NIST AI RMF, provides input to AI impact assessments, and links technical findings to enterprise risk appetite. Security managers are responsible for ensuring the process exists, is resourced, is repeatable and produces decisions that leadership can act upon.

Exam Tips: Answering Questions on Threat Modeling for AI Systems
• Think like a manager, not only an engineer. AAISM is a management-level certification. When multiple answers seem technically correct, prefer the one that emphasizes governance, structured process, risk-based prioritization, stakeholder involvement and alignment with business objectives.
• Remember that threat modeling is proactive and happens early. If a question asks when threat modeling should be performed, the best answer is during design and planning, and then continuously as the system changes. It is not primarily a post-incident activity.
• Know the lifecycle mapping. Expect questions that describe a threat and ask which lifecycle stage it targets. Data poisoning = training/data collection; adversarial examples and prompt injection = inference; model extraction = deployment/API exposure; supply chain = acquisition and development.
• Match the threat to the control. Memorize the pairings: poisoning to data validation and provenance; inversion/membership inference to differential privacy and output limiting; extraction to rate limiting and monitoring; prompt injection to input/output filtering and least privilege; supply chain to provenance verification and integrity checks.
• Distinguish STRIDE categories carefully. Tampering relates to integrity (poisoning), Information Disclosure to confidentiality (inversion, data leakage), Denial of Service to availability (resource exhaustion), Elevation of Privilege to excessive agency or tool misuse, Spoofing to identity of users or data sources, Repudiation to lack of logging and auditability.
• Recognize framework names and their purpose. MITRE ATLAS = AI adversary tactics and techniques; OWASP Top 10 for LLM = prioritized LLM vulnerabilities; NIST AI RMF = governance functions (Govern, Map, Measure, Manage); PASTA = risk-centric methodology; LINDDUN = privacy threats; attack trees = goal decomposition.
• Favor answers that start with system decomposition and asset identification. If asked what the first step is, it is almost always understanding the system and its trust boundaries, not selecting controls or running tools.
• Residual risk must be accepted by a risk owner. Watch for answers suggesting that security or the data science team alone decides to accept risk; the correct answer involves the appropriate business or executive owner.
• Beware of absolutes. Options claiming a control eliminates a threat, or that threat modeling is a one-time exercise, are usually wrong. AI risk is managed and reduced, not eliminated.
• Watch for third-party and supply chain clues. Whenever a scenario mentions a pre-trained model, external dataset, open-source library, plugin or API, the exam is likely testing supply chain threat modeling and the need to extend trust boundaries and perform provenance checks.
• Human oversight is a legitimate control. For high-impact or agentic AI scenarios, answers that include human-in-the-loop review or approval for consequential actions are often the best choice.
• Read scenario questions for the key qualifier. Words such as most effective, first, primary, best and most important signal that several options are plausible; select the one most directly tied to the stated objective and the manager's responsibility.
• Connect to monitoring and iteration. Questions on validating a threat model are answered by testing, red teaming, metrics and periodic review, not by assuming the design is complete.

Quick Review Summary
Threat modeling for AI systems is a structured, repeatable, lifecycle-wide process of decomposing the AI system, identifying assets and trust boundaries, enumerating threats using frameworks like STRIDE, MITRE ATLAS and OWASP, prioritizing by risk, selecting proportionate controls, validating effectiveness and iterating as the system and the threat landscape evolve. For the AAISM exam, demonstrate that you can map AI-specific threats to lifecycle stages and controls, apply the right framework, and always anchor decisions in governance, risk appetite and business objectives.