The AI Threat Landscape and Threat Actors

1. Why This Topic Matters

The ISACA Advanced in AI Security Management (AAISM) certification tests whether a security leader can govern, assess, and manage risk in organizations that build, buy, or use artificial intelligence. Within the AI Risk Management domain, understanding the threat landscape and the threat actors who populate it is foundational. You cannot perform a credible AI risk assessment, select proportionate controls, or brief a board on AI exposure unless you can articulate who might attack an AI system, why they would do so, what they would target, and how they would operate.

AI changes the threat picture in three distinct ways that exam writers expect you to recognize:

1. AI as a target – models, training data, pipelines, prompts, and outputs are new, often poorly protected assets.
2. AI as a weapon – adversaries use generative AI and machine learning to scale, automate, and personalize attacks (deepfakes, polymorphic malware, hyper-targeted phishing).
3. AI as an amplifier of existing weaknesses – AI systems inherit classic infrastructure, identity, data, and supply chain risks, and add new failure modes such as hallucination, bias, and emergent behavior.

From a management perspective, this topic ties directly to risk identification, risk appetite, threat modeling, control selection, incident response planning, and regulatory expectations (for example, the EU AI Act, NIST AI RMF, and ISO/IEC 42001 all require organizations to understand the risks and threats relevant to their AI systems).

2. What the AI Threat Landscape Is

The AI threat landscape is the complete set of potential adverse events, attack techniques, vulnerabilities, and adversaries that can compromise the confidentiality, integrity, availability, safety, fairness, and trustworthiness of AI systems and the business processes that depend on them. It spans the whole AI lifecycle: data collection, data preparation, model design, training, validation, deployment, inference (operation), monitoring, retraining, and decommissioning.

Key attack surfaces you must be able to name:

• Training data – sources, labeling processes, storage, and lineage.
• Models and weights – proprietary intellectual property, often stored as files that can be exfiltrated or tampered with.
• Pipelines and MLOps tooling – CI/CD for models, feature stores, notebooks, experiment trackers.
• Inference interfaces – APIs, chat interfaces, plugins, agents, tool integrations, retrieval-augmented generation (RAG) stores.
• Third-party components – pre-trained foundation models, open-source libraries, datasets, model hubs, and AI-as-a-Service providers.
• Humans – developers, data scientists, prompt engineers, end users, and the people who may be deceived by AI-generated content.

Reference frameworks commonly cited in AAISM-style material include MITRE ATLAS (Adversarial Threat Landscape for AI Systems, which maps adversary tactics and techniques against ML systems in a style similar to ATT&CK), the OWASP Top 10 for Large Language Model Applications, the NIST AI Risk Management Framework and its Generative AI Profile, ENISA threat landscape reports, and the NIST Adversarial Machine Learning taxonomy. You do not need to memorize every entry, but you should know what each framework is for and when a manager would use it.

3. Categories of AI-Specific Attacks (How the Threats Work)

A. Attacks on the training phase (poisoning and backdoors)
• Data poisoning – an adversary injects, alters, or mislabels training data so the model learns incorrect or malicious behavior. Can be availability poisoning (degrade overall accuracy) or targeted/backdoor poisoning (model behaves normally except when a hidden trigger appears).
• Model poisoning / supply chain tampering – a pre-trained model downloaded from a public hub contains a backdoor or malicious serialized code.
• Label flipping and clean-label attacks – subtle manipulations that are hard to detect with ordinary data quality checks.

B. Attacks on the inference phase (evasion and manipulation)
• Adversarial examples / evasion – carefully crafted inputs (imperceptible pixel changes, altered audio, malformed files) cause misclassification, for example bypassing malware detection or fraud models.
• Prompt injection – the signature threat for LLM applications. Direct prompt injection: the user overrides system instructions (jailbreaks, role-play exploits). Indirect prompt injection: malicious instructions are hidden in content the model processes (web pages, emails, documents in a RAG store), hijacking the model's behavior without the user's knowledge, which is especially dangerous for autonomous agents with tool access.
• Jailbreaking – coaxing a model into bypassing its safety alignment to produce harmful, prohibited, or confidential output.
• Sponge / resource exhaustion and model denial of service – inputs engineered to maximize compute cost, degrade latency, or run up API bills.

C. Attacks on confidentiality (extraction and inference)
• Model extraction / model theft – repeatedly querying a model to reconstruct a functional copy, stealing intellectual property and enabling offline attack development.
• Membership inference – determining whether a specific record was in the training set, which can expose sensitive personal data.
• Model inversion and training data extraction – reconstructing sensitive training inputs (faces, medical records, source code, credentials) from model outputs or memorized content.
• System prompt leakage – revealing proprietary instructions, hidden business logic, or embedded secrets.
• Sensitive information disclosure – the model repeats confidential data it was exposed to via RAG, fine-tuning, or prior conversation context.

D. Attacks via the AI supply chain and infrastructure
• Compromised open-source libraries and dependencies (classic software supply chain risk).
• Malicious or vulnerable third-party models, datasets, plugins, and agent tools.
• Insecure model serialization formats that execute code on load.
• Cloud misconfiguration exposing training buckets, vector databases, or API keys.
• Excessive agency – granting AI agents more permissions, tools, or autonomy than necessary, so a single manipulation produces real-world harm.

E. AI-enabled attacks against the wider organization
• Deepfakes (voice, video, image) used for CEO fraud, identity verification bypass, disinformation, and extortion.
• AI-generated social engineering – fluent, personalized, multilingual phishing and pretexting at scale.
• Automated reconnaissance and vulnerability discovery, AI-assisted exploit development, and adaptive, polymorphic malware.
• Disinformation and reputational attacks using synthetic content.
• Automated fraud against AI-driven decision systems (credit, insurance, hiring, content moderation).

F. Non-adversarial but security-relevant failure modes
Exam questions often blend intentional threats with unintentional ones: hallucination (confident false output), model drift, bias and discriminatory outcomes, over-reliance by users, shadow AI (employees using unsanctioned tools and leaking data), and insecure output handling (downstream systems trusting model output and executing it). Managers must treat these as risks even without a hostile actor.

4. Who the Threat Actors Are

Threat actors are classified by motivation, capability, resources, access, and intent. Expect questions that describe a scenario and ask you to identify the most likely actor or the control best suited to that actor's profile.

• Nation-state and state-sponsored actors – highly resourced, patient, and strategic. Motivations: espionage (stealing proprietary models, training data, and research), intellectual property theft, sabotage of critical AI-dependent infrastructure, influence operations using AI-generated disinformation, and gaining military or economic advantage. Capable of long-term supply chain compromise and insider recruitment.

• Organized cybercrime – financially motivated. Uses AI to industrialize phishing, business email compromise, deepfake fraud, ransomware negotiation, and credential stuffing; targets AI systems that gate money (fraud detection, payments, KYC/identity verification) with evasion attacks; steals models or API keys to resell or to run up costs.

• Hacktivists and ideologically motivated groups – seek publicity and disruption. Likely to jailbreak public chatbots to generate embarrassing output, deface or poison public-facing AI, expose bias, or leak training data to make a political point.

• Malicious insiders – data scientists, engineers, contractors, and administrators with privileged access to data, pipelines, and weights. Can poison data, exfiltrate models, insert backdoors, or abuse access to sensitive training records. Often the most damaging actor because they bypass perimeter controls and understand the system's weaknesses.

• Negligent or unwitting insiders – employees pasting confidential data into public generative AI tools, developers pulling unvetted models from public hubs, users over-trusting outputs. Not malicious, but a primary source of realized risk and a frequent exam distractor (the correct response is awareness, acceptable use policy, and technical guardrails rather than disciplinary action alone).

• Competitors and industrial espionage – motivated to steal models, prompts, and data for commercial advantage, often via model extraction or supply chain infiltration.

• Opportunistic attackers, script kiddies, and curious users – low sophistication but high volume. Share jailbreak prompts publicly, probe for prompt leakage, trigger abusive outputs, and exploit exposed endpoints with freely available tooling.

• Security researchers and red teamers – benign intent but can surface vulnerabilities publicly; organizations should channel them through coordinated disclosure and formal AI red-team programs.

• Third-party and supply chain actors – vendors, model providers, data brokers, and open-source maintainers whose compromise or poor practice becomes your exposure. Governance of these parties (due diligence, contractual controls, SBOM/AI-BOM, right to audit) is a management responsibility.

• Autonomous or semi-autonomous AI agents – an emerging category: agents (yours or someone else's) acting beyond intended boundaries, being hijacked via indirect prompt injection, or interacting unpredictably. Some frameworks treat misaligned or compromised AI as a threat source in its own right.

5. How Threat Analysis Works in Practice (The Management View)

AAISM is a management certification. The exam expects you to connect threat knowledge to a repeatable process:

1. Asset identification and AI inventory – you cannot assess threats to models you do not know exist. Maintain an inventory of AI systems, their data, criticality, and owners; address shadow AI.
2. Threat modeling – apply structured methods (STRIDE adapted for ML, MITRE ATLAS, OWASP LLM Top 10, attack trees) across the lifecycle. Ask: who would attack this, what would they gain, where could they enter, what would the impact be?
3. Threat intelligence – consume AI-specific intelligence (ATLAS case studies, vendor advisories, model hub security alerts, sector ISACs) and feed it into risk registers.
4. Risk assessment – combine threat likelihood (actor capability and motivation plus exposure) with impact (safety, financial, legal, reputational, privacy) to prioritize.
5. Control selection – map controls to threats: data provenance and validation against poisoning; input/output filtering, least-privilege tool access, and human-in-the-loop against prompt injection and excessive agency; rate limiting, monitoring, and watermarking against extraction; differential privacy and data minimization against inference attacks; vendor due diligence and model signing against supply chain risk; awareness and deepfake verification procedures against AI-enabled social engineering.
6. Adversarial testing and AI red teaming – validate assumptions by attacking your own systems before adversaries do.
7. Monitoring, detection, and incident response – log prompts and outputs, detect anomalous query patterns, define AI-specific incident playbooks (model rollback, prompt firewall updates, data quarantine).
8. Continuous review – the landscape evolves rapidly; threat assessments must be refreshed when models, data, use cases, or intelligence change.

6. Exam Tips: Answering Questions on The AI Threat Landscape and Threat Actors

Tip 1 – Think like a security manager, not a data scientist. ISACA questions favor answers involving governance, risk assessment, policy, accountability, and oversight over purely technical fixes. If one option says 'perform a threat and risk assessment of the AI system' and another says 'install an adversarial example detector,' the assessment is usually the better first step unless the question states the assessment has already been done.

Tip 2 – Watch for the key word: FIRST, BEST, MOST, PRIMARY. First typically points to understanding the context or assessing risk; best points to the most comprehensive, preventive, or risk-aligned option; primary asks for the fundamental purpose or the root concern.

Tip 3 – Match the attack to the lifecycle phase. Poisoning and backdoors belong to training; evasion, prompt injection, jailbreaks, and extraction belong to inference; dependency, model hub, and plugin risks belong to the supply chain. Many distractors describe a correct control applied to the wrong phase.

Tip 4 – Distinguish direct from indirect prompt injection. If the malicious instruction comes from the user, it is direct. If it arrives through retrieved documents, web pages, emails, or tool outputs, it is indirect. Indirect injection is the bigger concern for RAG systems and agents, and the best controls include treating all retrieved content as untrusted, privilege separation, and human approval for high-impact actions.

Tip 5 – Distinguish confidentiality attacks from each other. Model extraction steals the model's functionality; membership inference reveals whether a record was used in training; model inversion reconstructs the data itself; system prompt leakage exposes instructions. Questions about privacy regulations usually point to membership inference or inversion.

Tip 6 – Profile the actor from the clues. Long-term, stealthy, targeting research or critical infrastructure = nation-state. Fraud, ransomware, resale = organized crime. Publicity, embarrassing outputs = hacktivist or opportunist. Privileged access to weights or pipelines = insider. Data pasted into a public chatbot = negligent insider, answered with policy, awareness, and technical guardrails.

Tip 7 – Remember that AI is both target and weapon. If the scenario describes voice cloning, synthetic video, or hyper-personalized phishing, the threat is AI-enabled, and the answer lies in verification procedures, out-of-band confirmation, awareness training, and detection of synthetic media, not in hardening your own model.

Tip 8 – Do not forget non-adversarial risk. Hallucination, bias, drift, over-reliance, and insecure output handling are legitimate threats to trustworthiness. If a question describes harm with no attacker, do not force an adversary into the answer; choose validation, human oversight, monitoring, or output controls.

Tip 9 – Supply chain questions want due diligence and provenance. Vetting third-party models, verifying checksums and signatures, maintaining an AI bill of materials, contractual security requirements, and scanning serialized models are typical correct answers. 'Trust the vendor's documentation' is almost never correct.

Tip 10 – Know your frameworks by purpose. MITRE ATLAS = adversary tactics and techniques against AI/ML; OWASP LLM Top 10 = most critical application-level risks for LLM apps; NIST AI RMF = enterprise risk management structure (Govern, Map, Measure, Manage); ISO/IEC 42001 = AI management system; ENISA = European threat landscape reporting. Questions may ask which resource a manager would use to build an AI threat model (ATLAS or OWASP) versus a program (NIST AI RMF or ISO 42001).

Tip 11 – Least privilege and human-in-the-loop are frequent correct answers for agentic AI. When a scenario involves an AI system that can execute actions, send emails, or modify records, the strongest controls limit permissions, require approval for high-impact actions, and validate outputs before downstream use.

Tip 12 – Eliminate absolute and reactive answers. Options that 'completely eliminate' risk, 'block all AI use,' or rely solely on detection after harm are usually distractors. ISACA prefers risk-based, proportionate, preventive, and continuously monitored approaches aligned with the organization's risk appetite.

Tip 13 – Read the scenario for the real question. A long scenario about a chatbot leaking customer data may really be asking about data classification and access control for RAG sources, not about chatbot design. Identify the asset, the actor, the attack, and the control gap before choosing.

7. Quick Review Checklist

• Can you define the AI threat landscape across the full lifecycle?
• Can you name and differentiate poisoning, evasion, prompt injection (direct and indirect), jailbreaking, extraction, membership inference, inversion, denial of service, and supply chain attacks?
• Can you describe each major threat actor's motivation, capability, and typical targets?
• Can you explain AI as target versus AI as weapon?
• Can you map common controls to each threat category?
• Can you state what MITRE ATLAS, OWASP LLM Top 10, NIST AI RMF, and ISO/IEC 42001 are used for?
• Can you explain why threat assessments must be continuous and why shadow AI undermines them?

If you can answer each of these from memory and reason through a scenario by identifying the asset, actor, attack, phase, and proportionate management response, you are well prepared for AAISM questions on The AI Threat Landscape and Threat Actors.