AI Security Incident Definition and Classification

Why This Topic Matters
In the ISACA Advanced in AI Security Management (AAISM) body of knowledge, the domain of AI Governance and Program Management expects candidates to understand how an organization formally defines what counts as an AI security incident and how such incidents are classified by type, severity, and impact. Without a clear definition, organizations cannot detect, escalate, report, or learn from events that compromise AI systems. Traditional security incident taxonomies (malware, unauthorized access, data breach) do not fully capture AI-specific failure modes such as prompt injection, model poisoning, model theft, or harmful outputs. Regulators (for example under the EU AI Act, which requires reporting of 'serious incidents' for high-risk AI systems) and standards bodies (ISO/IEC 42001, NIST AI RMF) increasingly expect organizations to have documented incident criteria and classification schemes. On the exam, this topic tests whether you can distinguish an incident from an event or anomaly, map an AI-specific threat to a category, assign an appropriate severity, and recognize the governance artifacts (policy, taxonomy, escalation matrix) that must exist before an incident occurs.

What an AI Security Incident Is
An AI security incident is an event, or series of events, that actually or potentially compromises the confidentiality, integrity, availability, safety, or trustworthiness of an AI system, its data, its models, or the outcomes it produces, in violation of policy, law, or acceptable-use expectations. Key distinctions the exam expects you to know:

• Event – any observable occurrence in an AI system (e.g., a spike in rejected prompts). Not every event is an incident.
• Anomaly / Model drift – a statistical deviation in inputs or outputs. It becomes an incident only if it is caused by, or results in, a security or safety compromise, or if it crosses a defined threshold in policy.
• Security incident – an event confirmed (or reasonably suspected) to have an adverse impact requiring response.
• AI safety incident / harmful outcome – an incident where the AI produced outputs that caused or could cause harm (bias, unsafe advice, discriminatory decisions). Many frameworks treat these as a subset of AI incidents even when no classic 'attacker' exists.
• Near miss – a situation that could have become an incident but was prevented or did not materialize; mature programs track these for lessons learned.

The definition must be documented in policy, approved by governance (e.g., an AI governance committee or CISO), aligned with the enterprise incident response policy, and communicated to stakeholders including developers, data scientists, MLOps, legal, and business owners.

Why AI Incidents Need Their Own Definition
AI systems introduce failure modes not covered by traditional IT security:
• Attacks occur through data and inputs (poisoning, adversarial examples, prompt injection) rather than only through code vulnerabilities.
• Harm can arise with no attacker (hallucinations, bias, unsafe autonomous actions).
• Impact may be probabilistic and delayed – a poisoned model may behave correctly for months.
• The asset at risk includes model weights, training data, embeddings, prompts, and system prompts, each with distinct value and exposure.
• Third-party and supply-chain dependence (foundation models, APIs, datasets) blurs responsibility and detection boundaries.
• Regulatory reporting obligations may be triggered by AI-specific criteria (e.g., harm to health, safety, or fundamental rights).

How Classification Works
Classification is the structured process of assigning an incident to categories that drive response, escalation, and reporting. A typical AI incident classification scheme has several dimensions:

1. By Incident Type / Attack Category (taxonomy)
Commonly drawn from MITRE ATLAS, OWASP Top 10 for LLM Applications, and NIST AI RMF:
• Data poisoning – malicious or corrupted training or fine-tuning data altering model behavior.
• Model poisoning / backdoors – tampering with weights, checkpoints, or the training pipeline.
• Adversarial / evasion attacks – crafted inputs that cause misclassification or bypass controls.
• Prompt injection (direct and indirect) – manipulating LLM instructions via user input or embedded content.
• Jailbreaks / guardrail bypass – circumventing safety filters.
• Model extraction / theft – reconstructing or stealing model IP through queries or exfiltration of weights.
• Membership inference / data leakage / training data extraction – privacy compromises revealing sensitive training data.
• Sensitive information disclosure – model outputs leaking PII, secrets, or system prompts.
• Insecure output handling / excessive agency – model output executed downstream causing unauthorized actions.
• Supply chain compromise – malicious models, plugins, datasets, or libraries.
• Denial of service / resource exhaustion – sponge attacks, cost blowups, availability loss.
• Harmful, biased, or unsafe output – outcome-based incidents with ethical, legal, or safety consequences.
• Misuse / policy violation – unauthorized use (shadow AI), use outside intended purpose.

2. By Affected Asset or Lifecycle Stage
Data (collection, labeling), model (training, fine-tuning, storage), deployment (inference, APIs), and operations (monitoring, retraining). Knowing where in the AI lifecycle the incident occurred informs root-cause analysis and control improvement.

3. By Severity / Impact
Most organizations use tiers (e.g., Critical, High, Medium, Low or P1–P4) based on criteria such as:
• Scope and number of affected users, decisions, or records.
• Type of harm – safety, physical, financial, reputational, legal, fundamental rights.
• Sensitivity of data involved (PII, PHI, regulated data, trade secrets).
• Criticality of the AI system (high-risk per regulation, mission-critical business process).
• Degree of confirmed compromise versus suspected.
• Reversibility and duration of the impact.
• Regulatory reporting triggers (e.g., a 'serious incident' definition).

4. By Confidence / Status
Suspected, confirmed, contained, resolved, closed. This supports accurate metrics and avoids over- or under-reporting.

5. By Source
Internal detection (monitoring, red teaming), external report (user, researcher, vendor), regulator, or media.

Governance Components Supporting Definition and Classification
• AI incident response policy – states definitions, scope, roles, and authority; integrated with the enterprise IR policy.
• Incident taxonomy – the documented category list, mapped to recognized frameworks for consistency.
• Severity matrix and escalation paths – who is notified at each level (SOC, AI governance committee, legal, privacy officer, executive leadership, board).
• Decision criteria and playbooks – AI-specific playbooks (e.g., prompt injection, model rollback, dataset quarantine).
• Reporting obligations map – links classification outcomes to regulatory, contractual, and customer notification timelines.
• Roles and responsibilities – RACI including data scientists and MLOps engineers, who are often outside the traditional SOC.
• Metrics and continuous improvement – incident counts by category, mean time to detect/respond, near-miss tracking, and post-incident reviews feeding back into risk assessments and controls.

The Classification Process in Practice
1. Detection and triage – an alert or report is received; the triager determines whether it meets the AI incident definition (versus benign anomaly or normal drift).
2. Initial categorization – assign preliminary type and severity using the matrix; when in doubt, classify higher and downgrade later.
3. Escalation – notify stakeholders per the severity tier; engage legal/privacy early if data or regulatory impact is possible.
4. Investigation and reclassification – as facts emerge, refine type (e.g., from 'unexpected output' to 'indirect prompt injection') and severity.
5. Reporting – fulfill internal and external obligations based on final classification.
6. Lessons learned – update taxonomy, thresholds, and controls; record near misses.

Common Pitfalls
• Treating model drift or hallucinations as 'quality issues' and never routing them through incident management, so harmful-outcome incidents go unrecorded.
• Defining incidents only in terms of attackers, ignoring unintentional harm.
• Having no agreed threshold for when an anomaly becomes an incident.
• Failing to include third-party model or API failures in scope.
• Severity schemes that measure only technical impact and not harm to individuals or rights.
• Classification that is not linked to regulatory reporting timelines.

Exam Tips: Answering Questions on AI Security Incident Definition and Classification

Tip 1 – Know the hierarchy: event → anomaly → incident. If a question describes an observation with no confirmed or suspected adverse impact, the best answer usually involves investigating or monitoring, not declaring a major incident. If policy thresholds are crossed or compromise is suspected, it is an incident.

Tip 2 – Governance comes first. When asked what should be done FIRST or what is MOST important for effective AI incident management, favor answers about establishing a documented definition, taxonomy, severity criteria, and escalation paths approved by governance. ISACA consistently prioritizes policy and defined criteria over tools or ad hoc action.

Tip 3 – Match the scenario to the correct category. Learn the signature of each AI attack type:
• Manipulated training data → data poisoning.
• Hidden instructions in a web page or document the model reads → indirect prompt injection.
• User text overriding system instructions → direct prompt injection / jailbreak.
• Repeated queries used to rebuild the model → model extraction.
• Determining whether a record was in training data → membership inference.
• Model output triggering unauthorized downstream actions → insecure output handling / excessive agency.
• Tampered weights from a public repository → supply chain compromise.

Tip 4 – Severity is driven by impact and harm, not technical novelty. Choose the severity option that reflects scale, data sensitivity, potential harm to people, regulatory exposure, and business criticality. A low-complexity jailbreak that exposes PHI is a higher severity than a sophisticated attack on a non-critical internal demo.

Tip 5 – Remember that harm without an attacker still counts. Questions describing biased or unsafe outputs that caused harm are testing whether you recognize these as AI incidents requiring response and possibly reporting, not merely model performance issues.

Tip 6 – Link classification to reporting. If the scenario involves a high-risk system, personal data, or harm to individuals, the correct answer often includes engaging legal/privacy and assessing regulatory notification obligations as a direct result of classification.

Tip 7 – When unsure, escalate and reclassify later. ISACA favors conservative initial classification with downgrade after investigation over under-classifying and missing escalation windows.

Tip 8 – Integration, not isolation. The best answers align the AI incident process with the enterprise incident response program while adding AI-specific categories, roles (data scientists, MLOps), and playbooks – not a wholly separate, disconnected process.

Tip 9 – Watch for distractors. Options that focus on a single tool, blame an individual, or recommend immediately retraining the model without investigation are usually wrong. The exam rewards structured, repeatable, governance-driven decisions.

Tip 10 – Use the 'management' lens. AAISM is a management-level certification. Prefer answers that reference accountability, documented criteria, stakeholder communication, metrics, and continuous improvement over purely technical remediation details.

Quick Review Summary
• An AI security incident is an event that compromises or threatens the confidentiality, integrity, availability, safety, or trustworthiness of an AI system or its outcomes.
• Definitions must be documented, approved, and aligned with enterprise IR and regulatory requirements.
• Classification dimensions: type (taxonomy), affected asset/lifecycle stage, severity/impact, status, and source.
• Reference frameworks: MITRE ATLAS, OWASP LLM Top 10, NIST AI RMF, ISO/IEC 42001, EU AI Act serious-incident reporting.
• Severity is impact- and harm-based; classification drives escalation and reporting.
• Harmful outputs and near misses belong in scope.
• On the exam: define first, classify by impact, escalate conservatively, integrate with enterprise processes, and think like a manager.