AI Safety and Harm to Individuals and Society

Domain Context
This topic sits within the ISACA Advanced in AI Risk (AAIR) domain on AI Risk Governance and Framework Integration. It addresses how organizations identify, govern and mitigate the ways artificial intelligence systems can cause harm, both to specific people and to society at large. Within the AAIR body of knowledge, safety and harm are treated as a core driver of governance: the reason frameworks, risk appetites, controls and oversight structures exist in the first place is to prevent or reduce harm.

1. Why This Topic Is Important

AI systems differ from traditional IT systems in several ways that make harm more likely and more difficult to control:

- Scale and speed: a single flawed model can make millions of consequential decisions before anyone notices a problem.
- Opacity: many models are difficult to explain, so harmful behavior can go undetected or be impossible to justify to an affected individual.
- Probabilistic behavior: outputs are statistical rather than deterministic, so errors are expected rather than exceptional.
- Learned bias: models inherit and can amplify historical inequities present in training data.
- Autonomy: increasingly, AI acts without a human in the loop, removing a traditional safeguard.
- Dual use: capabilities built for legitimate purposes can be repurposed for fraud, surveillance, manipulation or attack.

From a governance perspective, harm matters because it creates legal liability (discrimination law, product liability, data protection, consumer protection, the EU AI Act), reputational damage, regulatory enforcement, loss of stakeholder trust and, in some sectors, physical danger. The AAIR exam expects you to understand that AI risk management is ultimately about protecting people, and that board-level accountability for AI extends to societal impact, not just enterprise loss.

2. What It Is: Defining AI Safety and Harm

AI safety is the discipline of ensuring that AI systems behave as intended, remain within acceptable boundaries, fail gracefully and do not cause unacceptable harm to people, property, the environment or society. Safety includes both intended-use safety (the system does its job reliably) and misuse and failure safety (the system resists abuse and degrades predictably).

Harm is any adverse impact on the rights, well-being, interests or opportunities of individuals, groups, organizations or society that is caused or contributed to by an AI system. Harm may be direct or indirect, immediate or delayed, intended or unintended, and may fall on people who never interact with the system (third parties).

2.1 Taxonomy of Harm to Individuals

- Physical harm: injuries from autonomous vehicles, robotics, medical AI misdiagnosis, industrial control systems.
- Psychological harm: manipulation, addictive design, exposure to harmful content, emotional dependence on chatbots, harassment via deepfakes.
- Financial and economic harm: unfair credit denial, algorithmic pricing discrimination, wrongful fraud flags freezing accounts, job loss from biased hiring tools.
- Discrimination and unfair treatment: disparate impact on protected groups in hiring, lending, housing, criminal justice, healthcare and insurance.
- Privacy harm: inference of sensitive attributes, re-identification, surveillance, unauthorized profiling, model memorization and leakage of personal data.
- Loss of autonomy and dignity: dark patterns, nudging, lack of meaningful consent, inability to contest an automated decision.
- Reputational harm: defamatory hallucinations, synthetic media depicting individuals falsely.
- Denial of access or opportunity: exclusion from services due to inaccessible design or biased eligibility models.

2.2 Taxonomy of Harm to Society

- Misinformation and disinformation: generative AI enabling cheap, scalable, persuasive false content.
- Erosion of democratic processes: micro-targeted political manipulation, synthetic candidates or statements, voter suppression.
- Systemic discrimination: entrenched inequities when biased models are deployed across entire sectors.
- Labor market disruption: displacement, deskilling, degradation of job quality, concentration of returns.
- Concentration of power: dependence on a small number of foundation model providers; market and informational monopolies.
- Environmental harm: energy and water consumption of training and inference, hardware supply chain impacts.
- Security and public safety: AI-enabled cyberattacks, bioweapon or chemical weapon uplift, autonomous weapons, critical infrastructure failures.
- Erosion of trust and epistemic harm: declining ability to distinguish authentic from synthetic information; the so-called liar's dividend.
- Cultural and linguistic homogenization: under-representation of minority languages and cultures in training data.
- Systemic and cascading risk: correlated failures when many institutions rely on the same model, data or vendor.

2.3 Key Related Concepts You Must Know

- Trustworthy AI characteristics (NIST AI RMF): valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed.
- Risk tiers (EU AI Act): unacceptable risk (prohibited practices such as social scoring, manipulative techniques exploiting vulnerabilities, certain biometric categorization and real-time remote biometric identification), high risk (employment, education, credit, law enforcement, critical infrastructure, medical devices, migration, justice), limited risk (transparency obligations such as disclosing chatbots and deepfakes), minimal risk.
- Human oversight models: human-in-the-loop (approves each decision), human-on-the-loop (monitors and can intervene), human-in-command (sets objectives and can shut down).
- Fundamental Rights Impact Assessment (FRIA) and Algorithmic Impact Assessment (AIA): structured assessments of how a system affects rights and groups before deployment.
- Red teaming: adversarial testing to discover harmful outputs, jailbreaks, bias and misuse pathways.
- Alignment: ensuring a system's objectives and behavior match human intent and values.
- Robustness: performance under distribution shift, adversarial inputs and edge cases.
- Fail-safe and fail-secure design: predictable, safe degradation when the system fails.
- Vulnerable populations: children, elderly, people with disabilities, economically disadvantaged and marginalized groups who face elevated or disproportionate harm.
- Contestability and redress: the ability for an affected person to understand, challenge and obtain correction of an AI-influenced decision.

3. How It Works: Governing Safety and Harm in Practice

The AAIR exam tests your ability to connect harm concepts to governance mechanisms. Think of the following lifecycle.

Step 1: Establish accountability and risk appetite
The board and senior management define the organization's AI principles, articulate a risk appetite for harm (often zero tolerance for certain categories such as unlawful discrimination or physical injury), assign ownership (for example a Chief AI Officer, AI ethics board or risk committee) and integrate AI into the enterprise risk management framework rather than treating it as a separate silo.

Step 2: Inventory and classify systems
Maintain an inventory of all AI systems, including third-party and embedded AI. Classify each by use case, affected population, decision consequence and regulatory tier. Classification determines the depth of assessment and control required. A system that influences access to employment, credit, healthcare or liberty is treated as high risk regardless of how simple the model is.

Step 3: Identify potential harms
Use structured methods: stakeholder mapping (who is affected, including non-users), harm taxonomies, scenario analysis, pre-mortems, consultation with affected communities, and review of incident databases. Consider intended use, reasonably foreseeable misuse and failure modes. Ask who benefits, who bears the risk and whether those are the same people.

Step 4: Assess likelihood and severity
Evaluate severity using dimensions such as reversibility, scale (number affected), breadth (groups affected), duration, and impact on fundamental rights. Combine with likelihood to prioritize. Severity of harm to individuals or society typically dominates: a low-probability, irreversible, rights-affecting harm is still unacceptable. Conduct an impact assessment (AIA, FRIA, DPIA) and document residual risk.

Step 5: Design and implement controls
Controls span the lifecycle:
- Data: representativeness analysis, data provenance, consent, minimization, bias audits of training data.
- Model: fairness metrics (demographic parity, equalized odds, calibration), robustness testing, adversarial testing, red teaming, safety fine-tuning, content filters, guardrails, output constraints.
- System: human oversight appropriate to risk, kill switches, rate limits, confidence thresholds that trigger escalation, fallback to manual processes.
- Process: pre-deployment approval gates, change management, model cards and system documentation, transparency notices, user disclosure of AI interaction, mechanisms for appeal and redress.
- Vendor: contractual safety requirements, right to audit, evaluation of foundation model provider safety practices and system cards.

Step 6: Monitor, detect and respond
Deploy continuous monitoring for drift, performance degradation across subgroups, anomalous outputs and user complaints. Establish AI incident response procedures, including severity classification, containment (pause or roll back), notification of affected individuals and regulators where required, root cause analysis and lessons learned. Participate in or consult incident repositories to learn from others.

Step 7: Assure and report
Perform independent reviews and audits of high-risk systems. Report to the board on harm metrics, incidents, near misses and emerging risks. Update risk appetite and controls as regulation and capabilities evolve.

3.1 Framework Integration
The AAIR exam emphasizes integrating multiple frameworks rather than choosing one:
- NIST AI RMF provides the Govern, Map, Measure, Manage functions and the trustworthiness characteristics.
- ISO/IEC 42001 provides an AI management system standard with organizational controls and impact assessment requirements; ISO/IEC 23894 provides AI risk management guidance aligned with ISO 31000.
- EU AI Act provides legally binding risk tiers, prohibited practices, high-risk obligations and transparency duties.
- OECD AI Principles and UNESCO Recommendation provide values-based reference points: human-centered values, fairness, transparency, robustness, safety and accountability.
- COBIT and ISACA guidance connect AI governance to enterprise IT governance, aligning AI objectives with stakeholder needs.
A mature organization maps its controls to all applicable frameworks, uses the strictest applicable requirement as the baseline and demonstrates traceability from principle to control to evidence.

4. How to Answer Exam Questions on This Topic

AAIR questions are scenario-based and test judgment, not memorization alone. Use this approach:

Identify the stakeholder at risk. Ask who is harmed: a user, a non-user third party, a protected group, or society. Answers that protect affected individuals and address rights generally outrank answers that only protect the organization.

Identify the type and severity of harm. Irreversible, rights-affecting, physical or large-scale harms demand the strongest response, often including pausing deployment. Reputational or efficiency concerns are secondary in the exam's hierarchy.

Look for the governance-level answer. When a question asks what should be done FIRST or what is MOST important, ISACA typically favors establishing accountability, performing an impact or risk assessment, or escalating to the appropriate governance body before implementing a technical fix.

Prefer proportionate, risk-based responses. Controls should match the risk tier. Blanket prohibition of all AI or ignoring low-risk systems are both usually wrong; the correct answer applies rigor where harm potential is highest.

Favor human oversight, transparency and redress. If an option provides affected individuals with notice, explanation and a route to contest decisions, it is frequently correct for questions about fairness, autonomy and accountability.

Recognize the role of the risk professional. The AI risk practitioner advises, assesses, facilitates and reports. Business owners own the risk; the board sets appetite. Options in which the risk function unilaterally makes business decisions or accepts risk on behalf of the enterprise are usually distractors.

Eliminate distractors. Common wrong answers: relying solely on vendor assurances, treating accuracy as sufficient evidence of safety, assuming removing a protected attribute eliminates bias (proxies remain), postponing assessment until after deployment, or assuming compliance equals ethics.

5. Exam Tips: Answering Questions on AI Safety and Harm to Individuals and Society

Tip 1 - People before profit in the answer hierarchy. If two options are plausible and one protects individuals or vulnerable groups while the other protects revenue or schedule, choose the former.

Tip 2 - Assess before you act. Questions that ask for the FIRST step almost always point to an impact assessment, risk assessment, stakeholder identification or classification activity rather than a control.

Tip 3 - Know the EU AI Act tiers. Be able to classify a scenario as prohibited, high risk, limited risk or minimal risk and recall the corresponding obligations, especially human oversight, data governance, documentation, logging, accuracy and robustness for high-risk systems, and disclosure duties for chatbots and synthetic content.

Tip 4 - Know the NIST trustworthiness characteristics. Questions may describe a failure and ask which characteristic was compromised. Safe relates to preventing harm to life, health, property or environment; valid and reliable relates to correct performance; fair relates to harmful bias.

Tip 5 - Severity dimensions matter. Reversibility, scale, breadth, duration and impact on fundamental rights determine severity. An irreversible harm to a small number of people may outrank a reversible inconvenience to millions.

Tip 6 - Proxies and disparate impact. Removing a sensitive attribute does not remove bias; correlated features act as proxies. Fairness must be tested on outcomes across groups, not assumed from inputs.

Tip 7 - Third parties and non-users count. Harm to people who never consented to or used the system, such as those depicted in deepfakes or screened by a hiring model, is still the deploying organization's responsibility.

Tip 8 - Human oversight must be meaningful. A rubber-stamp reviewer with no time, information or authority to override is not effective oversight. Expect questions on automation bias and over-reliance.

Tip 9 - Vendor and foundation model risk is still your risk. Using a third-party model does not transfer accountability for harm. Correct answers include due diligence, contractual safeguards, independent testing and monitoring of vendor models.

Tip 10 - Incident response applies to AI. When harm is detected, the sequence is typically contain (pause or roll back), assess, notify affected parties and regulators as required, remediate, and learn. Continuing to operate while investigating is rarely correct for high-severity harm.

Tip 11 - Compliance is a floor, not a ceiling. If a scenario describes something legal but clearly harmful or contrary to the organization's AI principles, the exam expects you to escalate and address the ethical risk, not simply note legal compliance.

Tip 12 - Watch for absolute language. Options containing always, never, all or eliminate are often distractors. Risk management reduces harm to acceptable levels; it rarely eliminates it entirely.

Tip 13 - Match the answer to the role in the question. If the question asks what the board should do, think oversight, appetite and accountability. If it asks what the risk practitioner should do, think assess, advise, report and escalate. If it asks what the system owner should do, think implement controls and own residual risk.

Tip 14 - Societal harm requires broader lenses. For misinformation, labor, environmental or democratic harms, correct answers often involve stakeholder engagement, transparency to the public, cross-industry collaboration, and alignment with external principles, not just internal controls.

Tip 15 - Document everything. Options involving documenting assessments, decisions, residual risk acceptance and testing results support accountability and are frequently part of the best answer.

6. Quick Revision Summary

- AI safety: behaving as intended, failing safely, resisting misuse.
- Harm to individuals: physical, psychological, financial, discriminatory, privacy, autonomy, reputational, exclusion.
- Harm to society: misinformation, democratic erosion, systemic bias, labor disruption, power concentration, environmental, security, trust erosion, cascading failure.
- Governance lifecycle: accountability and appetite, inventory and classify, identify harms, assess severity and likelihood, control, monitor and respond, assure and report.
- Frameworks: NIST AI RMF, ISO/IEC 42001 and 23894, EU AI Act, OECD, UNESCO, COBIT.
- Exam mindset: protect people first, assess before acting, proportionate controls, meaningful human oversight, transparency and redress, accountability never outsourced, compliance is the floor.

Master these concepts and the decision logic behind them and you will be well prepared for AAIR scenario questions on AI safety and harm to individuals and society.