Risk-Based AI Regulation and High-Risk Uses

Domain Context: AI Risk Governance and Framework Integration

This guide covers the concept of risk-based AI regulation and the identification and treatment of high-risk AI uses. This topic sits within the ISACA Advanced in AI Risk (AAIR) domain on AI Risk Governance and Framework Integration, where candidates are expected to understand how regulators classify AI systems by risk, what obligations attach to each tier, and how an organisation integrates these regulatory expectations into its existing governance, risk and compliance (GRC) structures.

1. Why This Topic Is Important

AI regulation around the world has converged on a single organising principle: the level of regulatory obligation should be proportionate to the level of risk the AI system poses to people, rights, safety and society. Rather than regulating the technology itself, lawmakers regulate the use case and its potential for harm.

For an AI risk professional this matters for several reasons:

- Legal exposure: Misclassifying a high-risk system as low risk can result in significant penalties. Under the EU AI Act, fines can reach up to 35 million EUR or 7% of global annual turnover for prohibited practices, and up to 15 million EUR or 3% for most other violations, including high-risk obligations.
- Resource allocation: A risk-based model allows organisations to concentrate control effort, assurance budget and oversight on the systems that matter most, rather than applying heavy controls uniformly.
- Framework integration: Risk tiers map naturally onto enterprise risk management (ERM) concepts such as inherent risk, risk appetite and control intensity, making it possible to embed AI regulation into COBIT, ISO/IEC 42001, ISO/IEC 23894 and the NIST AI Risk Management Framework (AI RMF).
- Trust and accountability: Demonstrating that high-risk uses are governed with human oversight, documentation and monitoring is central to building stakeholder trust and to satisfying auditors and regulators.

On the exam, this topic tests whether you can think like a regulator and a risk manager at the same time: identify the risk tier, name the obligations, assign them to the correct role, and integrate them into governance.

2. What Risk-Based AI Regulation Is

Definition: Risk-based AI regulation is a regulatory design approach in which AI systems are sorted into categories according to the severity and likelihood of harm they may cause, with legal requirements that escalate as the risk level rises. Systems with intolerable risk are banned; systems with high risk are tightly controlled; systems with limited risk carry transparency duties; and systems with minimal risk are largely unregulated beyond existing law.

The archetype: the EU Artificial Intelligence Act

The EU AI Act (Regulation (EU) 2024/1689) is the most frequently cited example and the model most exam questions will draw on. It establishes four principal risk tiers plus a separate regime for general-purpose AI (GPAI) models.

Tier 1 - Unacceptable risk (prohibited practices): AI uses banned outright because they conflict with fundamental rights and EU values. Examples include:
- Subliminal, manipulative or deceptive techniques that materially distort behaviour and cause significant harm.
- Exploitation of vulnerabilities related to age, disability or social or economic situation.
- Social scoring by public or private actors leading to detrimental or disproportionate treatment.
- Predictive policing based solely on profiling or personality traits.
- Untargeted scraping of facial images from the internet or CCTV to build facial recognition databases.
- Emotion recognition in workplaces and educational institutions (with narrow medical and safety exceptions).
- Biometric categorisation to infer sensitive attributes such as race, political opinions or sexual orientation.
- Real-time remote biometric identification in publicly accessible spaces for law enforcement, except in narrowly defined circumstances with authorisation.

Tier 2 - High risk: Permitted but subject to extensive obligations before and after being placed on the market. This is the core of the regulation and the focus of this guide (see Section 3).

Tier 3 - Limited risk (transparency obligations): Systems that interact with people or generate content must disclose their AI nature. Examples: chatbots must reveal they are not human; deepfakes and synthetic audio, image or video content must be labelled; AI-generated text published to inform the public on matters of public interest must be disclosed; emotion recognition and biometric categorisation systems must inform the people exposed to them.

Tier 4 - Minimal or no risk: The vast majority of AI, such as spam filters, recommendation engines in games or inventory optimisation. No new obligations beyond existing law, though voluntary codes of conduct are encouraged.

Parallel regime - General-purpose AI (GPAI) models: Foundation models carry their own duties regardless of downstream use: technical documentation, information for downstream providers, a copyright compliance policy, and a public summary of training content. GPAI models with systemic risk (presumed when training compute exceeds 10^25 FLOPs, or when designated by the Commission) face additional duties: model evaluation and adversarial testing, systemic risk assessment and mitigation, serious incident reporting, and cybersecurity protection.

Other jurisdictions and frameworks adopting a risk-based lens

- NIST AI RMF (United States): Voluntary, not a law, but risk-based through its four functions - Govern, Map, Measure, Manage - and its emphasis on context-specific risk tolerance. Often cited as the integration backbone for US-based organisations.
- US state laws: Colorado's AI Act (SB 24-205) targets high-risk AI systems that make or substantially influence consequential decisions in areas such as employment, housing, lending, education, healthcare and insurance, imposing duties of reasonable care to avoid algorithmic discrimination on both developers and deployers. New York City Local Law 144 requires bias audits of automated employment decision tools.
- Canada: The proposed Artificial Intelligence and Data Act (AIDA) focused on high-impact systems with risk assessment, mitigation and monitoring duties.
- United Kingdom: A principles-based, sector-regulator-led approach (safety, transparency, fairness, accountability, contestability) rather than a single statute, but regulators are expected to apply proportionate, risk-based oversight.
- OECD AI Principles and the Council of Europe Framework Convention on AI: Provide the risk-based, human-rights-centred values that national laws operationalise.
- ISO/IEC 42001 (AI Management System) and ISO/IEC 23894 (AI risk management guidance): Standards that give organisations a structured way to perform the impact assessments and controls that risk-based regulation requires.

Key terminology to master
- Provider: the entity that develops an AI system or has it developed and places it on the market or puts it into service under its own name or trademark.
- Deployer (called user in early drafts): the entity using an AI system under its authority in a professional context.
- Importer / Distributor: supply chain actors with verification duties.
- Conformity assessment: the process of demonstrating that a high-risk system meets the requirements, either through internal control or via a notified body.
- CE marking: affixed to compliant high-risk systems in the EU.
- Fundamental Rights Impact Assessment (FRIA): required of certain deployers (public bodies, and private entities providing public services, plus deployers of credit scoring and life/health insurance pricing systems) before using a high-risk system.
- Substantial modification: a change that alters compliance or intended purpose; can turn a deployer into a provider.

3. What Makes a Use High Risk

Under the EU AI Act there are two routes to high-risk classification.

Route A - Safety components of regulated products (Annex I): AI that is a product, or a safety component of a product, already covered by EU harmonisation legislation requiring third-party conformity assessment. Examples: machinery, toys, lifts, medical devices and in vitro diagnostics, civil aviation, motor vehicles, marine equipment, radio equipment, pressure equipment.

Route B - Stand-alone systems in listed use-case areas (Annex III): Eight domains where AI decisions significantly affect individuals:

1. Biometrics: remote biometric identification (where not prohibited), biometric categorisation by sensitive attributes, emotion recognition.
2. Critical infrastructure: safety components in the management of digital infrastructure, road traffic, and the supply of water, gas, heating and electricity.
3. Education and vocational training: determining access or admission, evaluating learning outcomes, assessing appropriate education level, monitoring prohibited behaviour during tests.
4. Employment and workers management: recruitment and selection (targeting job ads, filtering applications, evaluating candidates), decisions on promotion, termination, task allocation based on behaviour or traits, and monitoring and evaluating performance.
5. Access to essential private and public services: eligibility for public benefits, creditworthiness and credit scoring (except fraud detection), risk assessment and pricing in life and health insurance, triage of emergency calls and dispatch of emergency services.
6. Law enforcement: assessing risk of a person becoming a victim or offender, polygraphs, evaluating evidence reliability, profiling during investigations.
7. Migration, asylum and border control: risk assessments of individuals, examining applications, identifying people.
8. Administration of justice and democratic processes: assisting judicial authorities in researching and interpreting facts and law, and influencing election outcomes or voting behaviour.

The derogation (the exam loves this): An Annex III system is not high risk if it does not pose a significant risk to health, safety or fundamental rights because it only performs a narrow procedural task, improves the result of a previously completed human activity, detects decision-making patterns without replacing human assessment, or performs a preparatory task. However, any system that performs profiling of natural persons is always high risk. Providers relying on the derogation must document their assessment and register the system.

Generalised test for high-risk uses across frameworks

Whether or not the EU AI Act applies, exam questions frequently expect you to recognise high-risk characteristics in a scenario:
- The AI makes or materially influences a consequential decision about a person (hiring, credit, housing, healthcare, education, legal status, benefits).
- The affected population is vulnerable (children, patients, asylum seekers, people with disabilities).
- Harm would be severe, irreversible or at scale (safety-critical systems, infrastructure, mass biometric surveillance).
- The system uses sensitive data or infers protected attributes.
- There is limited human oversight, low transparency, or no realistic ability to contest the outcome.
- The system operates in a regulated sector with existing duties (financial services, health, employment).

4. How It Works: Obligations for High-Risk AI

Once a system is classified as high risk, a structured set of requirements applies. Candidates should be able to list them and attribute each to the correct role.

Provider obligations (Articles 9 to 17 of the EU AI Act, broadly):
- Risk management system (Art. 9): A continuous, iterative process across the entire lifecycle - identify foreseeable risks, estimate and evaluate them, adopt mitigation measures, and test. Residual risk must be acceptable, and special attention is given to impacts on children and vulnerable groups.
- Data and data governance (Art. 10): Training, validation and test datasets must be relevant, sufficiently representative, and as free of errors and complete as possible; the provider must examine possible biases and document data provenance, collection, labelling and cleaning choices.
- Technical documentation (Art. 11): Prepared before market placement and kept up to date; demonstrates compliance and enables authorities to assess the system.
- Record-keeping and logging (Art. 12): Automatic logging of events to enable traceability, post-market monitoring and incident investigation.
- Transparency and information to deployers (Art. 13): Instructions for use covering intended purpose, performance, limitations, known risks, human oversight measures, and expected lifetime and maintenance.
- Human oversight (Art. 14): Systems must be designed so that natural persons can understand capabilities and limitations, monitor operation, detect anomalies, decide not to use or override the output, and stop the system. For remote biometric identification, verification by at least two qualified persons is required before action is taken.
- Accuracy, robustness and cybersecurity (Art. 15): Appropriate levels declared in instructions; resilience to errors and adversarial attacks such as data poisoning, model poisoning, adversarial examples and model evasion; measures against feedback loops in continuously learning systems.
- Quality management system (Art. 17): Policies, procedures and instructions covering design, development, testing, data management, post-market monitoring and incident reporting.
- Conformity assessment, EU declaration of conformity and CE marking: Most Annex III systems use internal control; biometric systems may require a notified body; Annex I products follow their sectoral conformity route.
- Registration in the EU database for high-risk systems before placing on the market.
- Post-market monitoring and serious incident reporting: Serious incidents must be reported to market surveillance authorities, generally within 15 days, shorter for widespread infringements or death.
- Corrective actions and cooperation with competent authorities.

Deployer obligations (Art. 26 and 27):
- Use the system in accordance with instructions for use.
- Assign human oversight to competent, trained and authorised persons.
- Ensure input data is relevant and sufficiently representative where the deployer controls it.
- Monitor operation and inform the provider and authorities of risks or serious incidents; suspend use where risk is identified.
- Retain logs for at least six months (or longer where other law requires).
- Inform workers and their representatives before deploying high-risk AI in the workplace.
- Inform natural persons that they are subject to a high-risk AI system when it makes or assists decisions about them.
- Conduct a Fundamental Rights Impact Assessment where required.
- Comply with data protection obligations such as a DPIA under the GDPR.

When a deployer becomes a provider: If a deployer puts its name or trademark on a high-risk system, makes a substantial modification, or changes the intended purpose so that the system becomes high risk, the deployer assumes provider obligations. This is a common scenario-based exam question.

AI literacy (Art. 4): Both providers and deployers must ensure staff involved with AI systems have sufficient AI literacy. This applies to all AI, not only high-risk, and is often tested as a cross-cutting governance duty.

Implementation timeline (useful for scenario questions): The Act entered into force in August 2024. Prohibitions and AI literacy duties applied from February 2025; GPAI obligations and governance structures from August 2025; most high-risk (Annex III) obligations from August 2026; high-risk obligations for Annex I regulated products from August 2027.

5. How It Works: Integrating Risk-Based Regulation into Governance

The AAIR domain emphasises framework integration, meaning the exam will ask how the regulatory tiers become operational inside an organisation. A mature approach follows these steps.

Step 1 - Build and maintain an AI inventory. You cannot classify what you have not identified. The inventory should capture purpose, data, affected populations, jurisdictions, role (provider or deployer), vendor, and owner. Shadow AI and embedded AI in SaaS products must be included.

Step 2 - Classify each system by risk tier. Apply the regulatory criteria (prohibited, high, limited, minimal, GPAI) and the organisation's own impact criteria. Document the rationale, especially where the Annex III derogation is relied upon. Where multiple jurisdictions apply, classify to the strictest applicable standard or maintain jurisdiction-specific flags.

Step 3 - Map tiers to control intensity. High-risk systems receive full lifecycle controls (impact assessment, bias testing, human oversight design, documentation, logging, monitoring, independent assurance). Limited-risk systems receive transparency and disclosure controls. Minimal-risk systems follow baseline acceptable-use and security policies. This proportionality is the essence of risk-based governance.

Step 4 - Integrate with existing frameworks.
- NIST AI RMF: Map the Govern function to board oversight and policy; Map to inventory and classification; Measure to testing, bias evaluation and accuracy metrics; Manage to mitigation, monitoring and incident response. The RMF playbook can be used as a control catalogue to satisfy EU AI Act articles.
- ISO/IEC 42001: Use the AI management system as the quality management system required by Article 17; its Annex A controls (impact assessment, data governance, transparency, human oversight) align closely with high-risk obligations.
- ISO/IEC 23894 and ISO 31000: Provide the risk management process structure for Article 9.
- COBIT: Use governance and management objectives (for example EDM03 Ensured Risk Optimisation, APO12 Managed Risk, APO14 Managed Data) to anchor AI risk in enterprise IT governance.
- GDPR and privacy frameworks: Combine the DPIA and FRIA into a single integrated impact assessment to avoid duplication.
- Three lines model: First line owns the system and applies controls; second line (risk, compliance, privacy, legal) sets the classification methodology and challenges results; third line (internal audit) provides independent assurance that classification and controls are effective.

Step 5 - Govern the supply chain. Deployers depend on providers for documentation, performance claims and conformity. Contracts should allocate responsibilities, require instructions for use, guarantee cooperation for incidents, and address substantial modification. Vendor risk management must include AI-specific due diligence.

Step 6 - Monitor, report and reassess. Risk classification is not static. Changes in intended purpose, population, performance drift, new regulations, or incidents can shift a system into a higher tier. Establish key risk indicators, periodic reclassification reviews, incident escalation paths and board-level reporting.

Step 7 - Document everything. Regulators and auditors evaluate evidence. Classification decisions, derogation rationales, impact assessments, testing results, oversight arrangements and incident logs must be retained for the required periods (the Act requires providers to keep documentation for ten years after market placement).

6. Common Pitfalls and Misconceptions

- Believing the risk tier is determined by the technology (for example, all large language models are high risk). It is determined by the intended purpose and context of use.
- Assuming that being a deployer means having no obligations. Deployers carry significant duties, especially for high-risk systems.
- Confusing prohibited practices with high-risk uses. Prohibited uses cannot be mitigated with controls; high-risk uses can.
- Treating conformity assessment as a one-off event rather than a lifecycle commitment supported by post-market monitoring.
- Overlooking the profiling rule: profiling of natural persons in an Annex III area is always high risk, regardless of the derogation.
- Assuming the NIST AI RMF is mandatory. It is voluntary, though it may be contractually or sectorally required.
- Forgetting that limited-risk transparency duties (such as deepfake labelling) can apply simultaneously to a high-risk system.
- Ignoring extraterritorial reach: the EU AI Act applies to providers placing systems on the EU market and to providers and deployers outside the EU where the output is used in the EU.

7. Exam Tips: Answering Questions on Risk-Based AI Regulation and High-Risk Uses

Tip 1 - Identify the use case first, then the tier. When a scenario describes an AI system, ask: what decision does it make or influence, about whom, with what consequence? Then place it in the tier. Hiring, credit, education, law enforcement, critical infrastructure, biometrics, migration and justice are the high-risk signals. Social scoring, manipulation, untargeted facial scraping and workplace emotion recognition are prohibited signals. Chatbots and synthetic media without consequential decisions point to limited risk.

Tip 2 - Separate prohibited from high risk. If an answer option suggests adding controls to make a prohibited practice acceptable, it is wrong. Prohibited means stop. High risk means control.

Tip 3 - Know the role you are playing. Questions will often specify that the organisation is a deployer, a provider, or a deployer who has modified a system. Provider duties centre on design-time requirements (risk management system, data governance, technical documentation, conformity assessment, registration). Deployer duties centre on operational use (follow instructions, human oversight, log retention, inform affected people and workers, FRIA where applicable). If the deployer substantially modifies or rebrands the system, select the option that recognises it now bears provider obligations.

Tip 4 - Remember the always-high-risk profiling rule and the derogation conditions. If a question asks whether an Annex III system can be excluded from high-risk status, check for narrow procedural task, improving a human-completed result, pattern detection without replacing human assessment, or preparatory task - and remember profiling defeats the derogation.

Tip 5 - Choose proportionate answers. ISACA exams favour risk-based, proportionate responses. The best answer rarely applies maximum controls to every system or no controls to high-risk systems. Look for options that scale control intensity to the risk tier.

Tip 6 - Prefer governance and process over technology fixes. When asked for the best or first action, ISACA typically rewards establishing an inventory, classification methodology, accountability, policy or impact assessment before purchasing a tool or implementing a specific technical control.

Tip 7 - Integration questions want mapping, not replacement. If a question asks how to comply with multiple frameworks (for example EU AI Act, NIST AI RMF and ISO/IEC 42001), the correct answer usually involves mapping requirements into a unified control framework and single integrated impact assessment, rather than running parallel, duplicative programmes.

Tip 8 - Human oversight is a design requirement, not just a policy statement. If options differ between stating that humans will supervise versus designing the system so humans can understand, intervene, override and stop it, choose the latter.

Tip 9 - Watch for lifecycle and continuous language. Risk management systems, monitoring and reclassification are continuous. Answers describing one-time assessments at deployment are generally weaker than those describing ongoing processes.

Tip 10 - Know the key numbers but do not over-rely on them. Useful figures: four tiers plus GPAI; eight Annex III areas; 10^25 FLOPs systemic-risk presumption; fines up to 7% or 35 million EUR for prohibited practices and 3% or 15 million EUR for other violations; six-month minimum deployer log retention; ten-year provider documentation retention; 15-day serious incident reporting default; August 2026 as the main high-risk application date. Exam questions are more likely to test reasoning than exact figures, but these anchors help eliminate distractors.

Tip 11 - Read for extraterritoriality and jurisdiction. A US company whose AI output is used in the EU is in scope. A question that says the organisation has no EU presence may still require EU AI Act consideration if its system outputs are used in the Union.

Tip 12 - Eliminate absolutes. Options containing words like always, never, all AI systems or no obligations are frequently distractors, except where the Act genuinely uses absolutes (prohibited practices; profiling always high risk).

8. Quick Revision Summary

- Risk-based regulation scales obligations to potential harm: prohibited, high, limited, minimal, plus GPAI.
- High risk arises via Annex I regulated product safety components or Annex III use-case areas: biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration, justice and democracy.
- High-risk provider duties: risk management system, data governance, technical documentation, logging, transparency to deployers, human oversight, accuracy/robustness/cybersecurity, quality management, conformity assessment, CE marking, registration, post-market monitoring, incident reporting.
- High-risk deployer duties: follow instructions, trained human oversight, relevant input data, monitoring and reporting, log retention, inform workers and affected persons, FRIA where required.
- Deployers become providers through rebranding, substantial modification or change of intended purpose.
- Integration: inventory, classify, map tiers to control intensity, align with NIST AI RMF, ISO/IEC 42001, ISO/IEC 23894, COBIT and GDPR, embed in the three lines model, govern the supply chain, monitor and reassess continuously.
- Exam mindset: classify by use and context, distinguish prohibited from high risk, identify the role, choose proportionate and process-oriented answers, favour continuous lifecycle controls, and map frameworks rather than duplicate them.