Identifying AI Risk Scenarios

Overview
Identifying AI risk scenarios is a foundational task within the AI Risk Management for Security domain of the ISACA Advanced in AI Security Management (AAISM) certification. Before an organization can assess, treat, or monitor AI risk, it must first articulate what could go wrong, how, and with what consequences. A well-constructed risk scenario turns an abstract concern (for example, 'our chatbot might be attacked') into a concrete, analyzable statement that can be prioritized and acted upon. This guide explains why the topic matters, what an AI risk scenario is, how the identification process works, and how to answer exam questions on the subject.

Why Identifying AI Risk Scenarios Is Important
AI systems introduce risks that traditional IT risk registers were never designed to capture. Machine learning models learn from data rather than being explicitly programmed, so their behavior can be manipulated through the data pipeline, degraded over time, or exploited through inputs that look harmless to humans. If these failure modes are not identified up front, they will not be assessed, controls will not be designed for them, and the organization will be surprised by incidents it could have anticipated.

Key reasons the topic is emphasized on the exam:
- Completeness of the risk register: You cannot manage a risk you have not named. Scenario identification feeds every downstream step of the risk lifecycle.
- Alignment with business objectives: Scenarios link technical AI weaknesses to business impact, which is the language the board and executives understand.
- Regulatory expectation: Frameworks such as the NIST AI Risk Management Framework (Map function), ISO/IEC 23894, ISO/IEC 42001, and the EU AI Act all expect organizations to systematically identify AI-specific risks.
- Resource prioritization: Security budgets are finite. Clear scenarios allow leaders to focus controls on the most plausible and damaging events.
- Accountability: Each scenario can be assigned an owner, which supports governance and auditability.

What Is an AI Risk Scenario?
A risk scenario is a structured description of a potential event that, if it occurs, would affect the achievement of objectives. ISACA's Risk IT framework describes a scenario as being composed of several elements, and these translate directly to AI:

- Actor (threat source): Who or what initiates the event? Examples: an external attacker, a malicious insider, a careless data scientist, a third-party model provider, or a non-human cause such as data drift.
- Threat type: Malicious, accidental, failure, natural, or external requirement.
- Event: What actually happens? Examples: disclosure, modification, destruction, theft, inappropriate use, or ineffective design.
- Asset or resource: What is affected? In AI this includes training data, labeled datasets, feature stores, the model weights, the inference API, prompts and system instructions, embeddings, pipelines, and the decisions the model produces.
- Time dimension: Duration, timing (e.g., during training versus during inference), detection lag, and time to recovery.

A complete AI risk scenario therefore reads something like: 'An external attacker submits crafted inputs to the fraud-detection model's public API over several weeks, gradually learning the decision boundary and enabling fraudulent transactions to evade detection, resulting in financial loss and regulatory scrutiny.' Notice that this statement identifies actor, method, asset, event, and consequence.

Categories of AI-Specific Risk Scenarios
Candidates should be able to recognize and classify the major AI risk categories that commonly appear in scenarios:

1. Data poisoning: Attackers or faulty processes corrupt training data so the model learns incorrect or backdoored behavior. Occurs in the training phase.
2. Adversarial evasion: Inputs are subtly perturbed at inference time to cause misclassification (for example, altering an image so a malware classifier marks it benign).
3. Model extraction (model stealing): Repeated queries are used to replicate a proprietary model, creating intellectual-property loss.
4. Model inversion and membership inference: Attackers infer sensitive training data or determine whether a specific individual's record was used, causing privacy breaches.
5. Prompt injection and jailbreaking: For large language models, malicious instructions embedded in user input or in retrieved documents override system behavior, leading to data leakage or unauthorized actions by AI agents.
6. Hallucination and output integrity: The model produces confident but false outputs that are acted upon, causing operational, legal, or reputational harm.
7. Bias and fairness failures: Discriminatory outcomes lead to legal liability and loss of trust.
8. Model and data drift: Changes in the real-world environment degrade accuracy silently over time.
9. Supply chain risk: Compromised pre-trained models, poisoned open-source datasets, vulnerable ML libraries, or untrustworthy API providers.
10. Misuse and shadow AI: Employees use unsanctioned AI tools, pasting confidential data into public services.
11. Over-reliance and loss of human oversight: Automation bias leads humans to accept AI decisions without validation.
12. Excessive agency: Autonomous agents with broad permissions take harmful actions beyond their intended scope.
13. Availability and resource exhaustion: Denial of service against inference endpoints or runaway compute costs.
14. Compliance and legal exposure: Violations of privacy law, copyright, or sector-specific AI regulation.

How the Identification Process Works
Identifying AI risk scenarios is a repeatable process, not a one-time brainstorm. The typical steps are:

Step 1: Establish context and scope. Understand the business purpose of the AI system, its criticality, the data it uses, its users, its deployment environment, and the applicable legal and regulatory requirements. Risk appetite and tolerance set the lens through which scenarios will later be judged.

Step 2: Inventory AI assets. Maintain a register of AI systems, models, datasets, pipelines, third-party components, and integrations. Shadow AI must be discovered and included. You cannot write scenarios for assets you do not know exist.

Step 3: Map the AI lifecycle. Consider each phase: problem definition, data collection, data preparation, model training, validation, deployment, inference, monitoring, and retirement. Different risks arise at different stages. Poisoning belongs to training; evasion and prompt injection belong to inference; drift belongs to operations.

Step 4: Identify threat sources and vulnerabilities. Use structured references such as MITRE ATLAS (Adversarial Threat Landscape for AI Systems), the OWASP Top 10 for LLM Applications, the OWASP Machine Learning Security Top 10, and NIST AI 100-2 (Adversarial Machine Learning taxonomy). These provide vocabulary and coverage checks.

Step 5: Apply scenario-building techniques.
- Top-down: Start from business objectives and ask how AI could prevent their achievement.
- Bottom-up: Start from known technical weaknesses and generic scenario libraries and tailor them to the organization.
- Threat modeling: Adapt STRIDE or attack trees to the AI architecture, including data flows into and out of the model.
- Workshops and interviews: Bring together data scientists, security, legal, privacy, business owners, and ethics specialists. Cross-functional input is essential because AI risk spans disciplines.
- Lessons learned and incident data: Review internal incidents and public AI failure databases.
- Red teaming results: Adversarial testing surfaces realistic scenarios that desk analysis might miss.

Step 6: Document scenarios in a consistent format. Each entry should capture actor, threat type, event, affected asset, lifecycle phase, potential consequences (confidentiality, integrity, availability, safety, fairness, privacy, financial, reputational, legal), and an owner. Consistency makes later analysis comparable.

Step 7: Validate and refine. Check for relevance (is the scenario plausible for this organization?), completeness (are all lifecycle phases and asset types covered?), and granularity (not so broad it cannot be analyzed, not so narrow that the list becomes unmanageable). Remove duplicates and merge similar items.

Step 8: Hand off to analysis and maintain over time. Identified scenarios flow into likelihood and impact assessment. The list must be revisited when models are retrained, new use cases are added, the threat landscape changes, or regulations evolve.

Common Pitfalls
- Treating AI as 'just another application' and reusing a generic IT scenario list without AI-specific items.
- Focusing only on malicious attackers and ignoring accidental, failure-based, and drift-related scenarios.
- Omitting third-party and supply-chain scenarios because the model is 'vendor-managed.'
- Writing vague scenarios ('AI goes wrong') that cannot be assessed.
- Excluding business stakeholders, resulting in technically accurate scenarios with no stated business impact.
- Performing identification once and never updating it.

Exam Tips: Answering Questions on Identifying AI Risk Scenarios

Tip 1: Know where identification sits in the lifecycle. The exam frequently tests sequence. Identification comes after establishing context and asset inventory, and before analysis, evaluation, and treatment. If a question asks what should be done first, the answer is usually to understand the business context or inventory the AI assets rather than to select controls.

Tip 2: Recognize the components of a well-formed scenario. If asked which scenario statement is most useful or complete, choose the option that includes an actor, an event, an affected AI asset, and a consequence. Reject options that are merely a control, a vulnerability without an event, or a generic worry.

Tip 3: Match the risk to the lifecycle phase. Questions often describe a situation and ask what category of risk it represents. Remember: corrupted training data equals poisoning; manipulated inputs at runtime equal evasion or prompt injection; repeated queries to replicate behavior equal model extraction; inferring whether a person's data was used equals membership inference; declining accuracy over months equals drift.

Tip 4: Think like a manager, not only a technician. AAISM is a management-level certification. When two answers seem technically correct, prefer the one that emphasizes governance: involving stakeholders, aligning to business objectives, documenting in the risk register, assigning ownership, and using recognized frameworks.

Tip 5: Favor the comprehensive, structured approach. If options include an ad hoc brainstorm versus a structured method using threat libraries, lifecycle mapping, and cross-functional workshops, the structured approach is the intended answer.

Tip 6: Do not forget non-malicious scenarios. Distractors often focus exclusively on hackers. The best answers acknowledge accidental misuse, process failures, bias, drift, and vendor dependency as legitimate risk scenarios.

Tip 7: Separate identification from assessment and treatment. If a question asks about identifying scenarios, an answer that jumps to 'implement input validation' or 'calculate annual loss expectancy' is addressing a later phase and is likely a distractor.

Tip 8: Include third parties and shadow AI. When a question mentions externally sourced models, APIs, or employee use of public AI tools, expect the correct answer to extend scenario identification to those elements.

Tip 9: Watch for 'most' and 'best' qualifiers. Several options may be partially correct. Select the one that most directly produces a usable, business-relevant scenario or that most improves completeness of identification.

Tip 10: Use elimination. Discard options that are too narrow, that skip the business impact, that rely on a single person's judgment, or that treat identification as a one-time activity.

Worked Example
Question style: An organization is deploying a customer-service LLM that retrieves internal documents to answer questions. Which of the following would be the MOST effective way to identify relevant AI risk scenarios?
A. Review the vendor's security certification.
B. Conduct a cross-functional workshop that maps threats across the data retrieval, prompt handling, and output stages using an AI threat taxonomy.
C. Implement output filtering on the chatbot.
D. Calculate the expected financial loss from a data breach.

Reasoning: Option A is a single input and does not identify scenarios. Option C is a treatment. Option D is analysis. Option B is a structured, lifecycle-aware, multi-stakeholder identification activity using a recognized taxonomy. Answer: B.

Key Takeaways
- An AI risk scenario describes actor, event, asset, and consequence in a way that can be assessed.
- AI introduces unique scenarios across the lifecycle: poisoning, evasion, extraction, inversion, prompt injection, hallucination, bias, drift, supply chain, misuse, and excessive agency.
- Identification is structured, cross-functional, framework-informed, documented, and continuously updated.
- On the exam, choose answers that are comprehensive, governance-oriented, lifecycle-aware, and that stay within the identification phase rather than leaping to analysis or controls.