Adversarial Attacks: Evasion and Data Poisoning

1. Why This Topic Matters

Artificial intelligence systems introduce a new attack surface that traditional cybersecurity controls were never designed to protect. In classical IT security, an attacker targets the code, the network, or the credentials. In AI security, the attacker can target the learning process itself or the decision boundary the model has learned, without ever touching the underlying infrastructure. This is why the ISACA Advanced in AI Risk (AAIR) certification places adversarial attacks squarely within the AI Risk Program Management domain: a risk professional must be able to identify, assess, treat, and monitor these threats as part of an enterprise AI risk program, not as an afterthought bolted on by the data science team.

Adversarial attacks matter for several reasons that frequently appear in exam scenarios:
• Safety and physical harm – an evasion attack on an autonomous vehicle's vision model can cause it to misread a stop sign.
• Financial loss – poisoned fraud-detection models can be trained to ignore a specific pattern of fraudulent transactions.
• Regulatory exposure – the EU AI Act, NIST AI RMF, and ISO/IEC 42001 all expect organizations to address robustness, resilience, and data integrity.
• Erosion of trust – once a model is shown to be manipulable, stakeholders lose confidence in all outputs, not just the compromised ones.
• Supply chain risk – organizations increasingly fine-tune third-party foundation models or train on externally sourced datasets, inheriting any poisoning present upstream.

From an AAIR perspective, the key message is that adversarial attacks are a risk category that must be mapped to the AI risk register, assigned an owner, evaluated against risk appetite, and controlled across the full AI lifecycle.

2. What Adversarial Attacks Are

An adversarial attack is any deliberate action by a threat actor intended to cause a machine learning system to behave in a way that benefits the attacker and harms the system owner or its users. The two most heavily tested categories are evasion attacks and data poisoning attacks. They differ primarily in when in the lifecycle they occur and what they manipulate.

Evasion attacks (inference-time attacks)
Evasion attacks occur after the model has been trained and deployed. The attacker crafts an input – often called an adversarial example – that is deliberately perturbed so the model produces an incorrect output, while the input still looks normal to a human. The model itself is not changed; the attacker exploits weaknesses in the decision boundary the model already learned. Classic examples include:
• Adding imperceptible pixel noise to an image so a classifier labels a panda as a gibbon.
• Placing small stickers on a road sign so a vehicle perception system misreads it.
• Inserting unusual characters or synonyms into spam or malware so a filter classifies it as benign.
• Prompt injection against large language models, which many frameworks treat as a form of inference-time evasion or manipulation.

Data poisoning attacks (training-time attacks)
Data poisoning occurs before or during training (or during retraining and fine-tuning). The attacker inserts, modifies, or deletes training data so that the resulting model learns something wrong. Because the corruption is baked into the model's parameters, it persists until the model is retrained on clean data. Common forms include:
• Availability (indiscriminate) poisoning – degrades overall model accuracy so the system becomes unreliable.
• Targeted poisoning – causes misclassification only for specific inputs chosen by the attacker, while overall accuracy appears normal.
• Backdoor or Trojan attacks – the model behaves correctly unless a hidden trigger (a pattern, phrase, or watermark) is present, at which point it produces the attacker's chosen output.
• Label flipping – changing the labels on a subset of training examples.
• Clean-label poisoning – poisoned samples are correctly labeled and look legitimate, making them extremely hard to detect through label review.

Related terms you should recognize
• White-box attack – the attacker has full knowledge of the model architecture, parameters, and training data.
• Black-box attack – the attacker only has query access to inputs and outputs.
• Gray-box attack – partial knowledge.
• Transferability – adversarial examples crafted against one model often fool another model trained for the same task, enabling black-box attacks via a surrogate model.
• Model extraction / model stealing – repeatedly querying a model to reconstruct a copy, often a precursor to crafting evasion attacks.
• Model inversion and membership inference – privacy attacks that recover training data or determine whether a record was in the training set. These are often grouped with adversarial ML but are confidentiality attacks rather than integrity attacks.

3. How These Attacks Work

Mechanics of evasion
Machine learning models, especially deep neural networks, learn high-dimensional decision boundaries. These boundaries are not smooth in the way human perception is. Small, carefully computed changes to an input can push it across a boundary. Attackers typically use gradient-based optimization: they compute how the model's output changes with respect to each input feature and then adjust the input in the direction that maximizes error while keeping the total change tiny. Well-known techniques referenced in study materials include the Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini-Wagner attacks. In black-box settings, attackers either use query-based methods (observing output probabilities) or build a surrogate model and rely on transferability.

Mechanics of poisoning
Poisoning exploits the fact that models learn whatever statistical patterns exist in the training data. If an attacker can influence even a small percentage of that data, they can shift the learned boundary. Entry points include:
• Web-scraped or crowdsourced datasets with no provenance controls.
• User-feedback loops where production inputs are automatically fed back into retraining (for example, a recommendation engine or a spam filter that learns from user reports).
• Compromised data pipelines, data labeling vendors, or insider threats.
• Publicly shared pre-trained models or datasets on open repositories that have been tampered with.
Backdoor attacks are particularly dangerous because standard validation on a clean test set will not reveal them; accuracy metrics look healthy until the trigger appears in production.

Lifecycle view (critical for AAIR)
• Design and data collection – exposure to poisoning via untrusted sources.
• Training and validation – poisoning takes effect; evaluation may miss targeted or backdoor effects.
• Deployment and inference – evasion, prompt injection, model extraction.
• Monitoring and retraining – continuous learning loops reopen the poisoning window.
The exam expects you to connect each attack to the lifecycle stage where controls are most effective.

4. Risk Management and Controls

AAIR questions rarely ask you to implement a defense technically; they ask you to select the most appropriate governance or control response. Organize your thinking by control type.

Preventive controls
• Data provenance, lineage tracking, and integrity verification (hashing, signing) for training datasets.
• Vetting and contractual controls for third-party data providers, labeling vendors, and pre-trained models (AI supply chain risk management).
• Access control and segregation of duties over data pipelines and training environments.
• Data sanitization, outlier detection, and anomaly screening before training.
• Adversarial training – deliberately including adversarial examples during training to make the model more robust.
• Input validation, preprocessing, and feature squeezing at inference time.
• Rate limiting and query monitoring to hinder model extraction and query-based black-box attacks.
• Restricting or obfuscating confidence scores returned to users.

Detective controls
• Red teaming and adversarial robustness testing before deployment and periodically thereafter.
• Monitoring for distribution shift, abnormal input patterns, and unexpected prediction changes.
• Backdoor scanning and model behavior audits against trigger patterns.
• Logging of inference requests to support forensic investigation.

Corrective controls
• Incident response playbooks specific to AI (rollback to a known-good model version, retraining on verified data).
• Model versioning and the ability to revert quickly.
• Human-in-the-loop review for high-impact decisions.

Governance elements
• Inclusion of adversarial threats in the AI risk register and threat modeling (frameworks such as MITRE ATLAS and the NIST AI RMF Secure and Resilient characteristic are commonly referenced).
• Defined risk appetite and tolerance for model robustness.
• Clear ownership (model owner, data owner, security team) and reporting lines to the board or risk committee.
• Alignment with existing enterprise security and third-party risk programs rather than a separate silo.

5. How to Answer Exam Questions on This Topic

AAIR items are typically scenario-based. The question describes a situation and asks for the BEST, MOST effective, FIRST, or PRIMARY action, control, or risk. Use the following approach:

Step 1 – Identify the lifecycle stage. Ask yourself: is the attack happening to the training data (poisoning) or to a deployed model's inputs (evasion)? If the scenario mentions retraining on user feedback, web scraping, data vendors, or labeling, think poisoning. If it mentions manipulated inputs, stickers, perturbations, crafted queries, or prompts, think evasion.

Step 2 – Identify the risk property affected. Evasion and poisoning are primarily integrity attacks (the model produces wrong outputs). Model inversion and membership inference are confidentiality attacks. Availability poisoning and denial-of-service via expensive queries are availability attacks. Matching the attack to the CIA property often eliminates wrong answers.

Step 3 – Think like a risk manager, not an engineer. When choices include both a technical fix and a governance action, the exam usually favors the answer that addresses the root cause and fits a risk management process: perform a risk assessment, establish data provenance controls, update the risk register, define ownership, integrate into incident response. Purely technical answers are correct only when the question explicitly asks for a technical mitigation.

Step 4 – Apply the FIRST versus BEST logic. If asked what to do FIRST, choose assessment or identification activities (assess impact, identify affected models, verify the incident). If asked for the BEST long-term control, choose preventive and systemic measures (provenance controls, adversarial testing program, secure retraining pipeline).

Step 5 – Eliminate distractors. Common distractors include: encrypting the model (does not stop evasion), increasing model accuracy on the clean test set (does not detect backdoors), relying solely on a vendor's assurance (fails third-party risk principles), or retraining more frequently without securing the data source (increases poisoning exposure).

6. Exam Tips: Answering Questions on Adversarial Attacks: Evasion and Data Poisoning

• Timing is the key differentiator. Poisoning = training time, affects the model itself, persists until retrained. Evasion = inference time, affects individual predictions, model is unchanged. Memorize this distinction; it is the single most tested concept.

• Backdoors hide behind good metrics. If a scenario says the model performs well in validation but behaves strangely for certain inputs in production, the answer points toward a targeted poisoning or backdoor attack, and the appropriate control is backdoor testing, trigger scanning, or data provenance review, not more accuracy testing.

• Continuous learning equals continuous exposure. Any scenario with automated retraining on production or user-submitted data should trigger thoughts of poisoning risk; the best control is validating and curating data before it enters the training set.

• Transferability enables black-box attacks. If a question states the attacker has no access to the model internals yet still succeeds, the explanation is transferability or query-based attacks, and the mitigation includes query rate limiting and limiting exposed confidence scores.

• Adversarial training is a mitigation, not a cure. Questions may offer it as the single complete solution; the better answer usually combines adversarial training with monitoring, testing, and governance (defense in depth).

• Third-party and open-source models carry inherited risk. When the scenario involves downloading a pre-trained model or dataset, expect answers about supply chain due diligence, integrity verification, and independent testing before use.

• Map to frameworks when offered. Recognize NIST AI RMF functions (Govern, Map, Measure, Manage), MITRE ATLAS as the adversarial ML threat knowledge base, and ISO/IEC 42001 for management system requirements. The exam rewards knowing which framework serves which purpose.

• Human oversight is a legitimate control for high-impact decisions. For safety-critical or high-consequence use cases, a human-in-the-loop checkpoint is frequently the most appropriate compensating control when full technical robustness cannot be guaranteed.

• Read for qualifiers. Words such as imperceptible, crafted, perturbed, trigger, label, dataset, retrain, query, and surrogate are strong signals of the attack type the question is testing.

• Prefer answers that integrate AI risk into the enterprise program. ISACA consistently favors solutions that embed AI security into existing risk governance, incident response, and third-party management rather than standalone data science fixes.

7. Quick Reference Summary

Evasion: inference time; manipulates inputs; model unchanged; detect via input monitoring and robustness testing; prevent via adversarial training, input validation, rate limiting.
Data poisoning: training time; manipulates data; model corrupted; detect via data anomaly screening and backdoor testing; prevent via provenance, access control, supply chain vetting, secure retraining pipelines.
Governance anchor: include both in the AI risk register, threat model using MITRE ATLAS, align with NIST AI RMF and ISO/IEC 42001, assign ownership, test regularly, and maintain rollback capability.

Master the lifecycle distinction, think like a risk professional, and favor layered, governance-aligned controls, and you will handle AAIR questions on adversarial attacks with confidence.