Data Poisoning and Training Data Manipulation

Why This Topic Matters

Artificial intelligence systems are only as trustworthy as the data used to build them. Unlike traditional software, where logic is explicitly coded and can be reviewed line by line, machine learning models learn their behavior from training data. This creates a fundamental security exposure: if an adversary can influence what a model learns from, they can influence how the model behaves in production, often without touching the model code, the infrastructure, or the deployed system at all.

For the ISACA Advanced in AI Security Management (AAISM) certification, data poisoning sits squarely within the AI Risk Management for Security domain because it represents one of the most consequential and hardest-to-detect threats across the AI lifecycle. ISACA frames AI security as a governance and risk discipline, not just a technical one, so you are expected to understand data poisoning in terms of business impact, risk assessment, control selection, supply chain assurance, accountability, and continuous monitoring. The exam will test whether you can recognize the attack, identify where in the lifecycle it occurs, select the most appropriate control, and understand how it maps to frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001, MITRE ATLAS, and the OWASP Top 10 for LLM Applications.

Data poisoning is also important because its effects are persistent and latent. A poisoned model may pass standard accuracy testing, deploy successfully, and operate normally for months until a specific trigger activates malicious behavior. The cost of remediation is high because the typical fix is to identify and remove the contaminated data and retrain the model, which can be expensive, time-consuming, and in the case of large foundation models, practically infeasible for the downstream organization.

What Data Poisoning and Training Data Manipulation Are

Data poisoning is the deliberate corruption, injection, modification, or deletion of data used to train, fine-tune, validate, or continuously update a machine learning model, with the intent of degrading the model's performance, embedding hidden behaviors, or steering its outputs toward attacker-chosen results.

Training data manipulation is the broader category that includes poisoning but also covers tampering with labels, annotation pipelines, data preprocessing steps, feature engineering logic, sampling decisions, and feedback loops. The distinction the exam may draw is that poisoning typically refers to introducing malicious content, while manipulation can also include subtler alterations to how legitimate data is represented, weighted, or labeled.

Key characteristics you should remember:

- It is a training-time (or learning-time) attack, in contrast to evasion attacks and adversarial examples, which occur at inference time against an already-trained model.
- It targets the integrity of the model, and in availability-style variants, the availability or usefulness of the model.
- It exploits the data supply chain, which is often the weakest and least governed part of an AI system.
- It can be carried out by external attackers (who contaminate public or scraped data sources), insiders (data engineers, annotators, contractors), or compromised third parties (dataset vendors, labeling services, open-source dataset repositories, pre-trained model providers).

Categories of Data Poisoning Attacks

1. Availability (Indiscriminate) Poisoning
The attacker's goal is to degrade overall model performance so the model becomes unreliable or useless. Techniques include injecting large volumes of noisy or mislabeled data, flipping labels at random, or inserting contradictory examples. This is the AI equivalent of a denial-of-service attack. It is often easier to detect because accuracy on a clean validation set drops noticeably.

2. Targeted (Integrity) Poisoning
The attacker wants the model to misclassify or mishandle a specific input or class while otherwise performing normally. For example, causing a fraud detection model to approve transactions from a particular merchant, or causing a malware classifier to label one malware family as benign. Overall accuracy stays high, which makes it much harder to detect through standard testing.

3. Backdoor (Trojan) Poisoning
The attacker embeds a hidden trigger, such as a specific pixel pattern, a watermark, a rare token sequence, or a particular phrase, into training samples paired with an attacker-chosen label. The model learns to associate the trigger with the target output. In normal operation the model behaves correctly; when the trigger appears, the backdoor activates. This is the most insidious form because it is designed to survive validation. In LLMs, backdoors can be phrase-triggered behaviors such as emitting insecure code, bypassing safety guidelines, or producing biased content when a secret keyword appears.

4. Clean-Label Poisoning
The poisoned samples carry correct labels, so a human reviewer or label-consistency check will not flag them. Instead, the features of the samples are subtly perturbed so they shift the model's decision boundary. Because labels are legitimate, this attack defeats annotation audits and is a favored technique when the attacker can submit data but cannot control labeling.

5. Label Flipping
The attacker changes labels on a subset of legitimate samples, for example relabeling spam as not spam. This is simple, requires access to the labeling stage or dataset, and can be targeted or indiscriminate.

6. Data Injection and Data Modification
Injection adds new malicious records to the training set; modification alters existing records. Deletion attacks remove representative data to create blind spots, for example removing examples of a certain attack type from a security model's training data so the model never learns to recognize it.

7. Web-Scale and Supply Chain Poisoning
Modern foundation models are trained on scraped internet content, public datasets, and open repositories. Attackers can plant content on websites, Wikipedia-like sources, code repositories, or expired domains that are known to be crawled, so poisoned content ends up in future training corpora. Research has shown that controlling a surprisingly small fraction of a dataset can be enough to embed reliable backdoors. Supply chain poisoning also includes tampering with datasets hosted on public hubs, compromising labeling vendors, or distributing pre-trained models that already carry backdoors (sometimes called model poisoning or trojaned models).

8. Feedback Loop and Continuous Learning Poisoning
Systems that retrain on user interactions, telemetry, or human feedback are exposed to attackers who can submit malicious inputs at scale. Examples include manipulating recommendation systems, poisoning spam filters through coordinated false reports, or gaming reinforcement learning from human feedback (RLHF) by submitting adversarial ratings. The historical chatbot incidents in which public users trained a conversational agent to produce offensive content are a classic illustration.

9. Fine-Tuning and Retrieval Poisoning
Organizations frequently fine-tune third-party models on proprietary data or connect them to retrieval-augmented generation (RAG) knowledge bases. Poisoning the fine-tuning set or planting malicious documents in the retrieval corpus can alter model behavior. While RAG poisoning technically affects inference-time context rather than learned weights, the exam may group it under data manipulation because the control concepts (provenance, validation, access control over the knowledge base) are the same.

How Data Poisoning Works: The Attack Chain

Step 1 - Reconnaissance. The attacker identifies what data the target model uses, where it is sourced, how it is labeled, and whether the model retrains continuously. Public documentation, model cards, job postings, open-source pipelines, and vendor relationships all leak this information.

Step 2 - Gaining a Data Injection Point. This may be a public website that will be crawled, a user-generated content channel, a crowdsourced labeling platform, a compromised data vendor, insider access to a data lake, an unprotected storage bucket, or a public dataset repository that the organization pulls from without integrity verification.

Step 3 - Crafting Poisoned Samples. Depending on the goal, the attacker creates mislabeled samples, trigger-embedded samples, or clean-label perturbations. Sophisticated attackers optimize poisons so that a small number of samples has maximum effect and so that the poisons look statistically normal.

Step 4 - Insertion. Poisoned data enters the pipeline. Because data volumes are huge, a small contamination rate (often well under one percent) can be sufficient and is easily lost in the noise.

Step 5 - Training and Deployment. The model learns the malicious association. Standard metrics may look healthy, so the model is approved and deployed.

Step 6 - Exploitation. The attacker presents the trigger input or simply benefits from the degraded or biased behavior. For backdoors, this can be months or years after training.

Why Detection Is Difficult

- Training sets are too large for manual review.
- Targeted and backdoor attacks preserve overall accuracy.
- Clean-label attacks defeat label audits.
- Models are opaque, so it is hard to trace a behavior back to specific training samples.
- Organizations frequently lack data provenance, lineage, and integrity controls over training data.
- Downstream users of foundation models typically have no visibility into the original training corpus.

Business and Security Impacts

- Security control failure: Poisoned intrusion detection, fraud detection, malware classification, or content moderation models can be made blind to specific threats.
- Safety and harm: Autonomous systems, medical diagnosis tools, or hiring tools making harmful decisions.
- Bias and discrimination: Deliberately injected bias can create legal and regulatory exposure under anti-discrimination and AI regulations.
- Reputational damage: Models producing offensive, false, or manipulated outputs.
- Financial loss and fraud: Manipulated credit, trading, pricing, or recommendation models.
- Regulatory non-compliance: Failure to demonstrate data governance and integrity under the EU AI Act, ISO/IEC 42001, or sector regulations.
- Loss of trust in AI programs: Undermining organizational confidence in AI adoption.

Risk Management Approach (The ISACA Lens)

ISACA expects candidates to treat data poisoning as a risk to be governed across the lifecycle rather than a purely technical bug. A structured approach includes:

Identify. Inventory all AI systems and their data sources. Classify data by sensitivity and criticality. Document data lineage: where it came from, who touched it, how it was transformed. Maintain an AI bill of materials (AIBOM) covering datasets, pre-trained models, and third-party components.

Assess. Conduct threat modeling specific to the data pipeline. Evaluate likelihood based on exposure (public scraping, user-generated data, third-party vendors, continuous learning) and impact based on the criticality of model decisions. Use MITRE ATLAS to enumerate relevant adversary techniques. Assess third-party and open-source dataset providers through due diligence.

Treat. Select controls proportionate to risk (see below). Apply defense in depth: preventive controls on data ingestion, detective controls during training and operation, and corrective controls for recovery.

Monitor. Continuously monitor model behavior, data drift, and anomalies. Establish incident response playbooks specific to suspected poisoning. Report metrics to governance bodies.

Key Controls and Mitigations

Data Governance and Provenance
- Maintain documented data lineage and chain of custody for all training data.
- Use trusted, vetted data sources; apply vendor risk management to dataset and labeling providers.
- Require cryptographic hashing, digital signatures, or checksums to verify dataset integrity when sourcing from external repositories.
- Create data cards and model cards documenting origin, collection method, and known limitations.

Access Control and Pipeline Security
- Enforce least privilege and role-based access to data lakes, feature stores, labeling tools, and training pipelines.
- Apply separation of duties between those who collect data, those who label it, and those who approve training runs.
- Protect data at rest and in transit; secure storage buckets and MLOps infrastructure.
- Log and audit all changes to training datasets; use immutable or versioned data storage so tampering can be detected and rolled back.

Data Validation and Sanitization
- Perform statistical analysis to detect outliers, distribution shifts, duplicate injections, and anomalous clusters.
- Apply label-consistency checks, cross-validation of annotators, and consensus labeling for critical datasets.
- Use anomaly detection and data-cleaning techniques to filter suspicious samples before training.
- Limit the influence of any single data source or contributor (rate limiting, per-source caps) in crowdsourced or continuously learning systems.

Robust Training Techniques
- Use robust statistics and training methods that reduce sensitivity to small numbers of outliers.
- Apply ensemble methods or bagging so that no single poisoned subset dominates.
- Consider differential privacy techniques, which limit the influence of individual samples on the model and incidentally reduce poisoning effectiveness.
- Use data augmentation and regularization to reduce overfitting to trigger patterns.

Model Testing and Verification
- Test against a held-out, highly trusted golden validation set that is protected separately from the training pipeline.
- Perform adversarial testing and red teaming specifically for backdoors, including trigger-search and neuron-activation analysis techniques.
- Compare model behavior across versions to detect unexpected changes.
- Conduct independent model validation before promotion to production, consistent with model risk management practices.

Operational Monitoring and Response
- Monitor production outputs for drift, anomalous predictions, and unusual confidence patterns.
- Maintain the ability to roll back to a known-good model and dataset version.
- Develop incident response procedures for suspected poisoning, including forensic analysis of data lineage, model retraining from verified data, and stakeholder notification.
- For continuous learning systems, implement human-in-the-loop review before new data influences the production model, or use staged retraining with validation gates.

Third-Party and Foundation Model Risk
- Perform due diligence on model providers regarding their training data governance and poisoning defenses.
- Prefer providers that publish transparency documentation and undergo independent assessment.
- Fine-tune only with verified internal data; treat RAG knowledge bases as production assets with change control.
- Recognize residual risk: downstream organizations cannot fully verify upstream training data, so compensating controls (output filtering, monitoring, guardrails) are required.

Framework Mapping You Should Know

- NIST AI RMF: Data poisoning relates to the Map and Measure functions (identifying and measuring risks to validity, reliability, security, and resilience) and the Manage function (treating risks). Data integrity is a component of the trustworthiness characteristic 'Secure and Resilient'.
- NIST AI 100-2 (Adversarial Machine Learning taxonomy): Classifies poisoning attacks by attacker goal (availability, integrity, privacy), capability, and knowledge; distinguishes poisoning (training-time) from evasion (inference-time).
- MITRE ATLAS: Includes techniques for poisoning training data and publishing poisoned datasets or models, under tactics such as Resource Development and ML Attack Staging.
- OWASP Top 10 for LLM Applications: Lists Training Data Poisoning (LLM03 in the 2023 version) and Data and Model Poisoning (LLM04 in the 2025 version), and Supply Chain Vulnerabilities (LLM03 in 2025) as closely related risks.
- ISO/IEC 42001 and ISO/IEC 23894: Require AI management system controls covering data quality, data provenance, and AI risk management across the lifecycle.
- EU AI Act: High-risk AI systems must meet data governance requirements, including examination of data for biases and appropriate data quality criteria, and must be resilient against attempts to manipulate training data.

Distinguishing Data Poisoning from Related Attacks

Exam questions often test whether you can tell closely related concepts apart:

- Data poisoning manipulates training or learning data; it happens before or during training and alters the model itself.
- Evasion attacks / adversarial examples manipulate inputs at inference time to fool a fixed model; the model is not changed.
- Prompt injection manipulates instructions given to an LLM at runtime; no retraining occurs. Indirect prompt injection via retrieved documents can resemble RAG poisoning but targets instruction following, not learned weights.
- Model inversion and membership inference are confidentiality attacks that extract information about training data from the model; they do not alter the model.
- Model theft / extraction copies model functionality; it is a confidentiality and IP attack.
- Model poisoning / trojaned models deliver an already-compromised model artifact; this is closely related and is often grouped with data poisoning as a supply chain risk.
- Data drift is a natural, non-malicious change in data distribution over time; it can look similar in monitoring but has no adversary. Questions may ask you to distinguish accidental data quality issues from deliberate manipulation.

Roles and Accountability

ISACA emphasizes governance. Expect questions about who is responsible for what:
- The board and senior management set risk appetite for AI and ensure resources for AI security.
- The AI governance committee or AI risk owner ensures data poisoning is in the risk register and controls are proportionate.
- Data owners and data stewards are accountable for data quality, provenance, and access decisions.
- ML engineers and data scientists implement validation, robust training, and testing.
- Security teams secure the pipeline, monitor for anomalies, and lead incident response.
- Internal audit provides independent assurance over data governance and model validation controls.
- Procurement and third-party risk management assess dataset vendors and model providers.

Exam Tips: Answering Questions on Data Poisoning and Training Data Manipulation

Tip 1 - Identify the lifecycle stage first. If the scenario describes manipulated inputs to an already-deployed model, the answer is evasion or prompt injection, not poisoning. If the scenario involves training data, labeling, fine-tuning datasets, retraining on user feedback, or data sourced from the internet or third parties, think poisoning. Always ask yourself: does this attack change what the model learns, or just what it sees at runtime?

Tip 2 - Recognize the symptom patterns. A model that performs well in testing but fails only on specific inputs or under a specific pattern points to a backdoor or targeted poisoning. A model whose overall accuracy has dropped suggests availability poisoning or possibly non-malicious data quality problems. Correctly labeled but subtly altered samples point to clean-label attacks. Mislabeled samples point to label flipping.

Tip 3 - Favor preventive, governance-level controls as the BEST answer. ISACA exams consistently prefer answers that address root cause through governance, process, and preventive control over purely reactive or technical fixes. When asked for the best or most effective control, options like establishing data provenance and integrity verification, enforcing access controls and separation of duties over the data pipeline, or implementing a data governance framework usually beat options like retraining the model or increasing monitoring alone. Monitoring is a detective control; it is important but is rarely the best answer when a preventive option is available.

Tip 4 - Watch for the word FIRST. When a question asks what should be done first, the answer is usually about understanding or assessing: inventory data sources, perform a risk assessment, conduct threat modeling, or identify the data lineage. Do not jump to implementing a technical control before the risk is understood.

Tip 5 - Supply chain questions want third-party due diligence plus compensating controls. If the scenario involves a pre-trained model, a public dataset, or an outsourced labeling service, the best answers involve vendor risk assessment, integrity verification of acquired artifacts, contractual requirements, and compensating controls such as independent validation and output monitoring. Remember that you cannot fully verify a foundation model's training data; the exam will reward recognizing residual risk and the need for compensating controls.

Tip 6 - Continuous learning scenarios call for human review and validation gates. If a system retrains on user interactions, the exam expects controls like rate limiting per contributor, anomaly detection on incoming data, staged retraining with a protected validation set, and human-in-the-loop approval before data influences production.

Tip 7 - Know the impact language. Data poisoning is primarily an integrity attack. Availability poisoning affects availability. It is generally not a confidentiality attack; if a question describes extraction of training data, that is model inversion or membership inference.

Tip 8 - Golden validation sets must be protected separately. A frequent distractor is to validate a model using data from the same pipeline that could be poisoned. The correct approach is an independently curated, access-controlled, trusted test set.

Tip 9 - Remember the small-percentage principle. If an answer choice implies that poisoning requires controlling a large share of the data, it is likely wrong. Effective backdoors can be achieved with a very small fraction of the dataset.

Tip 10 - Map to frameworks when asked. Be ready to associate poisoning with NIST AI RMF trustworthiness characteristics (secure and resilient, valid and reliable), MITRE ATLAS poisoning techniques, OWASP LLM Data and Model Poisoning, and the data governance requirements of ISO/IEC 42001 and the EU AI Act.

Tip 11 - Eliminate distractors systematically. Typical wrong answers include: controls that address a different attack (input filtering for a poisoning scenario, encryption for an integrity problem), controls that are purely reactive, controls that are disproportionate to the risk, or answers that place accountability with the wrong role (for example making the data scientist accountable for data ownership rather than the data owner).

Tip 12 - Think like a risk manager, not just an engineer. The AAISM exam is a management-level certification. When two answers seem technically valid, choose the one that reflects a structured, risk-based, lifecycle-wide approach aligned to organizational governance and accountability.

Sample Question Walkthroughs

Scenario 1: A security operations team notices that its ML-based malware classifier correctly identifies all malware families in testing but consistently labels files containing a particular unusual byte string as benign. What type of attack is MOST likely?
Analysis: Normal overall performance plus a specific trigger causing misclassification is the signature of a backdoor (trojan) poisoning attack. Evasion is a distractor because the behavior is consistent and tied to a learned trigger, not a one-off crafted input.

Scenario 2: An organization plans to fine-tune a third-party foundation model using a dataset downloaded from a public repository. Which control would BEST reduce the risk of training data poisoning?
Analysis: Options such as monitoring outputs after deployment are detective and reactive. The best answer is to verify the integrity and provenance of the dataset (for example, through cryptographic verification against a trusted source) and validate the data before use, which is preventive and addresses the root cause.

Scenario 3: A recommendation system retrains nightly on user clicks. Management is concerned that coordinated bot activity could manipulate recommendations. What should the AI security manager recommend FIRST?
Analysis: Before selecting controls, the manager should assess the risk and threat model of the feedback pipeline. If the question instead asks for the BEST control, favor anomaly detection on incoming interaction data combined with per-source influence limits and a validation gate before retraining.

Scenario 4: Which role is accountable for ensuring the quality and provenance of data used to train a high-risk AI system?
Analysis: The data owner holds accountability. Data scientists and engineers are responsible for implementation but are not the accountable party in ISACA's governance model.

Summary

Data poisoning and training data manipulation are training-time integrity attacks that exploit the data supply chain to corrupt what a model learns. They range from crude label flipping to sophisticated clean-label backdoors that survive standard validation and activate only when triggered. Because effects are latent, persistent, and expensive to remediate, the ISACA AAISM exam expects you to approach the topic as a lifecycle risk management problem: identify data sources and lineage, assess exposure, prefer preventive governance and pipeline controls, protect independent validation data, apply rigorous third-party due diligence, monitor continuously, and maintain tested response and rollback capabilities. In exam questions, locate the lifecycle stage, match the symptom to the attack type, prefer preventive and governance-level answers, assess before acting when asked what to do first, and keep accountability with the correct role.