AI Incident Definition and Classification

Why This Topic Matters

Organizations are deploying AI systems at scale, and when those systems fail, behave unexpectedly, or cause harm, the consequences can be severe: regulatory penalties, reputational damage, financial loss, and real harm to individuals. Yet many organizations still rely on traditional IT incident processes that were never designed to recognize when a model begins to drift, produce discriminatory outputs, hallucinate false information, or be manipulated through prompt injection. Without a clear definition of what constitutes an AI incident and a consistent classification scheme, incidents go undetected, are escalated inconsistently, and cannot be learned from. For the ISACA Advanced in AI Risk (AAIR) certification, this topic sits within the AI Risk Program Management domain because incident definition and classification are foundational to detection, response, reporting, and continuous improvement of the AI risk program. Regulators such as those enforcing the EU AI Act also impose serious incident reporting obligations, making a well-defined taxonomy a compliance necessity rather than a nice-to-have.

What Is an AI Incident?

An AI incident is an event in which the development, deployment, or use of an AI system directly or indirectly leads to actual harm, or a near miss that could plausibly have led to harm, to people, organizations, property, the environment, or fundamental rights. This definition is intentionally broad and aligns with frameworks such as the OECD AI Incidents Monitor and the NIST AI Risk Management Framework. Key elements of the definition include:

• Involvement of an AI system – the AI component (model, data pipeline, inference service, agent) is a contributing cause, not merely present in the environment.
• Harm or potential harm – physical, psychological, financial, reputational, legal, societal, or rights-based harm.
• Deviation from expected behavior – performance degradation, unsafe outputs, unauthorized actions, or violation of policy, law, or ethical commitments.
• Near misses and hazards – mature programs also capture events that did not cause harm but exposed a vulnerability. An AI hazard is a condition that could lead to an incident; distinguishing hazards from incidents is a common exam point.

It is important to differentiate an AI incident from related concepts:

• Traditional IT/security incident – e.g., a server outage hosting a model. It becomes an AI incident only if the AI behavior itself contributed to harm.
• Model performance issue – accuracy dropping below a threshold is a hazard or minor incident unless it produces harmful decisions.
• Ethical concern or complaint – a stakeholder complaint may trigger an investigation that confirms or rules out an incident.

Categories of AI Incidents

Classification typically begins with type. Common AI incident categories include:

• Performance and reliability failures – model drift, data drift, concept drift, degraded accuracy, hallucinations, unsafe or nonsensical outputs.
• Bias, fairness, and discrimination – disparate impact on protected groups, unfair denial of services, biased rankings or recommendations.
• Safety incidents – physical harm from autonomous systems, robotics, medical AI, or control systems.
• Security and adversarial incidents – prompt injection, jailbreaking, data poisoning, model inversion, membership inference, model theft or extraction, evasion attacks.
• Privacy and data protection incidents – leakage of personal or sensitive training data, unauthorized inference of attributes, improper data collection.
• Transparency and accountability failures – undisclosed AI use, inability to explain a consequential decision, missing audit trails.
• Misuse and abuse – AI used for deepfakes, fraud, disinformation, harassment, or outside its intended purpose.
• Compliance and legal incidents – violations of the EU AI Act, GDPR, sector regulations, intellectual property rights, or contractual obligations.
• Third-party and supply chain incidents – failures in vendor models, APIs, foundation models, or datasets inherited by the organization.
• Operational and governance failures – deployment without approval, bypass of human oversight, lack of monitoring, unauthorized model changes.

How Classification Works

Classification is a structured process that converts a raw report of an event into an actionable record. A robust scheme usually classifies along several dimensions:

1. Incident type – using the taxonomy above to establish the nature of the failure.

2. Severity – the magnitude of actual or potential impact. Typical levels are Critical, High, Medium, and Low (or Severity 1 through 4). Severity criteria should consider:
• Number of individuals or stakeholders affected (scale)
• Type of harm (physical safety and fundamental rights typically rank highest)
• Reversibility (irreversible harm is more severe)
• Financial and legal exposure
• Regulatory reporting triggers (e.g., EU AI Act "serious incident" criteria such as death, serious health damage, serious disruption of critical infrastructure, or breach of fundamental rights)
• Reputational and public trust impact

3. Likelihood of recurrence – whether the root cause is systemic or isolated.

4. Lifecycle stage – whether the incident originated in data collection, training, validation, deployment, monitoring, or decommissioning. This supports root cause analysis.

5. Root cause category – data quality, model design, inadequate testing, human error, adversarial action, inadequate oversight, vendor failure.

6. Affected stakeholders – customers, employees, the public, vulnerable groups, regulators, business partners.

7. Detection source – automated monitoring, user complaint, internal audit, red teaming, media, regulator, or whistleblower.

8. Urgency and escalation path – determined by combining severity and other factors, this dictates who must be notified (AI governance committee, legal, privacy officer, CISO, executive management, board) and within what timeframe.

A practical classification matrix combines impact and likelihood (similar to a risk heat map) to produce a severity rating that drives response. Many organizations extend their existing incident management framework (e.g., ITIL or NIST SP 800-61) with AI-specific triggers, fields, and escalation rules rather than building a parallel process. This integration ensures AI incidents feed into enterprise risk registers, business continuity plans, and board reporting.

The Incident Lifecycle and the Role of Classification

• Detection and reporting – monitoring tools (drift detection, output guardrails, anomaly detection), user feedback channels, and internal reporting mechanisms surface potential incidents. A clear definition lowers the threshold for reporting and encourages a no-blame culture.
• Triage and classification – the incident is confirmed, categorized, and assigned severity. Misclassification at this stage is the most common cause of delayed or inadequate response.
• Containment and response – actions scale with severity: rollback to a previous model version, disabling the feature, adding human-in-the-loop review, blocking malicious inputs, or shutting down the system.
• Notification – internal escalation plus external notification where required (regulators, affected individuals, customers, partners). Classification determines whether regulatory clocks start ticking.
• Investigation and root cause analysis – tracing the failure through the AI lifecycle.
• Remediation and recovery – retraining, data correction, control redesign, policy updates.
• Lessons learned and continuous improvement – updating the taxonomy, thresholds, monitoring, and training based on trends. Aggregated classification data enables metrics such as incident frequency by type, mean time to detect, and mean time to resolve.

Governance Considerations

The AI risk program should document the incident definition and classification scheme in policy, assign clear ownership (often a cross-functional AI incident response team including data science, legal, privacy, security, compliance, and business owners), define Service Level Agreements for each severity level, and test the process through tabletop exercises. The scheme should be reviewed periodically as regulations evolve and as the organization's AI portfolio changes. Alignment with recognized frameworks (NIST AI RMF Manage function, ISO/IEC 42001, ISO/IEC 23894, OECD AI incident definitions) strengthens defensibility and supports audits.

Exam Tips: Answering Questions on AI Incident Definition and Classification

Tip 1 – Know the broad definition. If a question asks what qualifies as an AI incident, favor answers that include actual harm and potential harm (near misses), and that span harms beyond cybersecurity, such as discrimination, rights violations, and safety. Answers that limit AI incidents to security breaches or outages are usually distractors.

Tip 2 – Distinguish hazards, near misses, and incidents. A discovered vulnerability with no exploitation is a hazard; an event that almost caused harm is a near miss; an event that caused harm is an incident. Questions often test whether you know mature programs track all three.

Tip 3 – Severity is driven by impact, not technical complexity. When asked to rank or prioritize incidents, choose the option affecting safety, fundamental rights, vulnerable populations, irreversible harm, or large numbers of people. A low-accuracy internal recommendation engine is less severe than a biased lending model affecting thousands of applicants.

Tip 4 – The first step in response is classification and triage, not remediation. ISACA questions frequently ask what should be done "first" or "next." Confirming the incident, classifying it, and assessing impact typically precede containment, notification, and root cause analysis. Notification to regulators generally comes after the incident is confirmed and classified as meeting reporting thresholds.

Tip 5 – Look for integration, not isolation. The best answer usually integrates AI incident handling into the enterprise incident and risk management frameworks while adding AI-specific criteria, rather than creating a completely separate process or relying solely on the data science team.

Tip 6 – Remember the regulatory angle. Expect questions referencing serious incident reporting under the EU AI Act or privacy breach notification laws. Classification schemes must identify reportable incidents and the associated timelines. The reason to classify quickly is often to meet these obligations.

Tip 7 – Think cross-functionally. Questions about who should be involved in defining or classifying AI incidents favor cross-functional governance (legal, privacy, security, ethics, business owners, data science) over a single technical function.

Tip 8 – Root cause mapping to the lifecycle. If a question describes drift after deployment, poisoned training data, or inadequate validation, identify the lifecycle stage; the correct classification and remediation often depend on it.

Tip 9 – Watch for "most" and "best" qualifiers. ISACA exams reward the answer that provides the greatest risk reduction or governance value. A documented, consistently applied classification taxonomy with defined severity thresholds and escalation paths beats ad hoc judgment every time.

Tip 10 – Continuous improvement is the end goal. If an answer option mentions using classified incident data to update risk assessments, monitoring thresholds, training, or the taxonomy itself, it reflects program maturity and is often correct.

Sample Question Walkthrough

An organization's customer service chatbot begins providing incorrect refund policy information to a small number of users, and one user posts the exchange on social media. Which of the following should the AI risk team do FIRST?

A. Notify the regulator of a serious incident
B. Retrain the model on corrected policy documents
C. Confirm the event, classify its type and severity, and assess impact
D. Issue a public apology

The correct answer is C. Classification and impact assessment come first because they determine whether regulatory notification is required (option A is premature and the event may not meet serious incident thresholds), what containment and remediation are appropriate (option B is a remediation step), and what communications are warranted (option D is a response step). This pattern—classify before you act—is central to answering AAIR questions in this area correctly.

Key Takeaways

• An AI incident is any event where an AI system contributes to actual or potential harm across a wide range of harm types.
• Classification covers type, severity, lifecycle stage, root cause, stakeholders affected, detection source, and escalation path.
• Severity is based on impact factors such as scale, type of harm, reversibility, and regulatory thresholds.
• A well-defined scheme integrates with enterprise incident management, supports regulatory reporting, and feeds continuous improvement of the AI risk program.
• On the exam, classify first, prioritize by impact, think cross-functionally, and choose answers reflecting documented, integrated, and continuously improved processes.