Data Quality, Integrity and Validation Controls

1. Why This Topic Matters

Artificial intelligence systems are fundamentally shaped by the data used to train, tune, test and operate them. The well-known principle garbage in, garbage out applies with greater force to AI than to traditional software, because a model does not merely process data – it learns from it. Errors, bias, gaps, duplication, stale records or deliberately corrupted inputs do not stay isolated; they are absorbed into model weights and then reproduced, at scale, in every prediction the model makes.

For the ISACA Advanced in AI Security Management (AAISM) exam, this topic sits inside the AI Technologies and Controls domain. ISACA expects candidates to understand that data controls are a first line of defense for AI security, not an afterthought. Poor data quality creates business risk (wrong decisions, regulatory fines, discriminatory outcomes), while compromised data integrity creates security risk (data poisoning, backdoors, model manipulation). Validation controls are the mechanisms that detect and prevent both.

Key reasons the exam emphasizes this area:
• Risk amplification – a single corrupted data source can degrade an entire model and every downstream system that consumes its outputs.
• Regulatory expectations – the EU AI Act, ISO/IEC 42001, NIST AI RMF and sector regulations explicitly require data governance, data quality assessment and documentation of training data.
• Security attack surface – data pipelines are a prime target for adversaries (poisoning, label flipping, supply-chain manipulation of public datasets).
• Trust and accountability – you cannot demonstrate a fair, explainable or reliable model if you cannot demonstrate the quality and integrity of the data behind it.

2. What It Is – Core Definitions

Data Quality is the degree to which data is fit for its intended purpose. ISACA and ISO 8000 / DAMA-DMBOK describe quality through a set of dimensions that you should memorize:
• Accuracy – data correctly reflects the real-world entity or event.
• Completeness – required values are present; no critical gaps or excessive nulls.
• Consistency – the same fact is represented the same way across systems and over time.
• Timeliness / Currency – data is sufficiently up to date for the use case.
• Validity / Conformity – data conforms to defined formats, ranges, types and business rules.
• Uniqueness – no unintended duplicates.
• Relevance – data is appropriate and representative for the AI task (closely tied to bias and fairness).
• Representativeness – the dataset reflects the population and conditions the model will face in production.

Data Integrity is the assurance that data has not been altered, destroyed or corrupted in an unauthorized or accidental manner across its lifecycle. It is the I in the CIA triad. Integrity has several facets:
• Physical integrity – protection against hardware failure, bit rot, storage corruption.
• Logical integrity – entity integrity (unique keys), referential integrity (valid relationships), domain integrity (allowed values).
• Provenance and lineage integrity – you can trace where data came from, who touched it and how it was transformed.
• Adversarial integrity – protection against intentional tampering such as data poisoning, label manipulation or injection of backdoor triggers.

Validation Controls are the checks, rules and procedures that verify data meets quality and integrity requirements before it is used, while it is processed and after it is produced. In AI contexts this includes input validation at ingestion, dataset validation before training, validation of model inputs at inference time and validation of outputs before they are consumed by people or downstream systems.

A useful mental model: Quality asks is the data good enough; Integrity asks is the data what it is supposed to be; Validation is how we prove both.

3. How It Works – Controls Across the AI Data Lifecycle

The exam frequently frames questions by lifecycle stage. Learn which controls belong where.

3.1 Data Sourcing and Acquisition
• Source vetting and trust assessment – evaluate reputation, licensing, update cadence and security of third-party or open datasets. Public datasets are a known supply-chain poisoning vector.
• Data contracts and SLAs – formal agreements defining schema, quality thresholds, refresh frequency and change notification between data producers and AI teams.
• Provenance capture – record origin, collection method, consent basis, timestamps and responsible owner. Tools: datasheets for datasets, data cards.
• Legal and ethical screening – confirm rights to use the data, personal data minimization and that collection did not introduce sampling bias.

3.2 Ingestion and Pipeline Controls
• Schema validation – enforce expected columns, data types, nullability and encodings; reject or quarantine nonconforming records.
• Range, format and business-rule checks – e.g., age between 0 and 120, dates not in the future, currency codes from an approved list.
• Checksums and cryptographic hashes – verify files have not changed in transit or at rest; digital signatures for datasets from trusted providers.
• Secure transport and storage – TLS in transit, encryption at rest, write-once or immutable storage for golden training sets.
• Deduplication and reconciliation – detect duplicate or conflicting records; reconcile record counts and control totals between source and destination.
• Access control and segregation of duties – only authorized roles may modify training data; separate those who prepare data from those who approve it.

3.3 Preparation, Labeling and Feature Engineering
• Data profiling – statistical summaries (distributions, null rates, cardinality, outliers) establish a quality baseline.
• Cleansing rules with audit trails – imputation, outlier handling and normalization must be documented and reproducible.
• Label quality assurance – inter-annotator agreement, gold-standard samples, double labeling, review of annotator performance. Label flipping is a classic poisoning technique.
• Bias and representativeness analysis – check demographic or class distributions against the target population; document known gaps.
• Data versioning – treat datasets like code (e.g., DVC, lakeFS, feature stores) so every model can be traced to an exact data snapshot.
• Train/validation/test separation – prevent leakage; ensure test data is not contaminated by training data.

3.4 Training-Time Integrity Controls
• Anomaly and outlier detection on the training set – flag records statistically distant from the rest, which may indicate poisoning.
• Poisoning defenses – data sanitization, robust statistics, influence-function analysis, trimmed training, spectral signature detection, ensemble disagreement.
• Dataset hashing and attestation – hash the final training set and record it in the model card or model registry so retraining can be verified.
• Reproducibility checks – retraining with the same data and seed should yield comparable results; divergence signals integrity issues.

3.5 Inference-Time and Operational Controls
• Input validation at the API or interface – type, length, encoding and content checks; reject malformed inputs; this also mitigates prompt injection and adversarial inputs for LLMs.
• Data drift and concept drift monitoring – compare live input distributions with training distributions (population stability index, KL divergence, KS test); trigger review or retraining when thresholds are breached.
• Output validation – range checks, confidence thresholds, guardrails, human-in-the-loop for high-impact decisions.
• Feedback-loop protection – if production outputs are recycled as future training data, validate them first to prevent self-reinforcing errors or attacker-seeded corruption.
• Logging and monitoring – record data access, modifications and validation failures; integrate with SIEM for security alerting.

3.6 Governance Layer (Spanning All Stages)
• Data governance roles – Data Owner (accountable, defines quality requirements and classification), Data Steward (responsible for quality rules and remediation), Data Custodian (implements technical controls and storage).
• Data quality metrics and KPIs/KRIs – e.g., percent of records failing validation, null rate, duplicate rate, drift score, time to remediate.
• Policies and standards – classification, retention, acceptable sources, labeling standards, minimum quality thresholds before training is approved.
• Documentation – datasheets, model cards, lineage diagrams; required by ISO/IEC 42001 and expected under the EU AI Act for high-risk systems.
• Independent assurance – internal audit or third-party review of data pipelines and controls.

4. Mapping to Frameworks the Exam May Reference

• NIST AI RMF – the MAP and MEASURE functions call for understanding data provenance, assessing data quality and measuring bias and drift.
• ISO/IEC 42001 (AI Management System) – Annex A controls on data for AI systems: data acquisition, data quality, data provenance and data preparation.
• ISO/IEC 5259 series – data quality for analytics and machine learning.
• ISO 8000 – general data quality standard.
• COBIT – Managed Data (APO14) addresses data quality strategy, assessment and cleansing.
• EU AI Act Article 10 – high-risk systems require training, validation and testing data subject to data governance practices, relevance, representativeness and examination for bias.
• OWASP Top 10 for LLM Applications – Training Data Poisoning and Supply Chain Vulnerabilities are directly related.
• MITRE ATLAS – catalogs adversarial techniques including poisoning of training data.

5. Threats and Failure Modes to Recognize

• Data poisoning – attacker injects crafted records to degrade accuracy or implant a backdoor triggered by specific inputs.
• Label flipping – deliberately mislabeling records to shift decision boundaries.
• Supply-chain compromise – tampered public datasets, pre-trained models or data-processing libraries.
• Data leakage – test data or target variable information leaking into training, creating falsely high performance.
• Silent schema change – upstream system changes a field meaning or unit, causing systematic error.
• Drift – real-world conditions change, so once-valid data no longer represents reality.
• Bias through unrepresentative sampling – quality issue that becomes an ethical and legal issue.
• Feedback contamination – model outputs re-enter the training set without validation (model collapse risk for generative AI).

6. How to Answer Exam Questions on This Topic

AAISM questions are scenario-based and ask for the best, most effective, first or primary answer. Use the following reasoning approach:

Step 1 – Identify whether the problem is quality, integrity or validation. If the scenario describes inaccurate, incomplete or outdated data, think quality dimensions and governance. If it describes unauthorized change, tampering or poisoning, think integrity controls (hashing, access control, provenance, anomaly detection). If it asks how to detect or prevent bad data entering a stage, think validation controls at that specific stage.

Step 2 – Locate the lifecycle stage. The best control is usually the one closest to where the risk originates. Bad third-party data: source vetting and data contracts. Corrupted files: hashes and signatures. Mislabeled data: labeling QA. Changing live inputs: drift monitoring.

Step 3 – Prefer preventive over detective, and detective over corrective, unless the question specifically asks how to detect or respond.

Step 4 – Prefer governance and root-cause answers when the question is about a manager or program-level decision. AAISM is a management certification. If a choice says define data quality requirements with the data owner or establish data governance roles, it often beats a purely technical fix when the scenario is about accountability or repeated issues.

Step 5 – Prefer risk-based answers. Not all data needs the same rigor. Controls should be proportionate to the criticality of the AI use case and the sensitivity of the data.

Step 6 – Eliminate distractors. Common distractors include: retraining the model (does nothing if the data remains bad), increasing model complexity, adding more data of unknown quality, encrypting data (confidentiality, not quality), and penetration testing the application (does not address data pipeline integrity).

7. Exam Tips: Answering Questions on Data Quality, Integrity and Validation Controls

• Tip 1 – Know the quality dimensions cold. Expect a question that describes a symptom and asks which dimension is affected. Duplicate customers = uniqueness; wrong postal codes = accuracy; missing income fields = completeness; last refreshed two years ago = timeliness; different date formats across systems = consistency; values outside allowed list = validity.

• Tip 2 – Integrity equals unauthorized or accidental change. If the stem mentions tampering, poisoning, unauthorized modification or corruption, the answer involves hashing, digital signatures, immutable storage, access control, segregation of duties, lineage or anomaly detection – not data cleansing.

• Tip 3 – Validation belongs at boundaries. The most effective place for validation is at the point of entry: ingestion for training data, API input for inference, and before consumption for outputs. If asked where to place a control, choose the earliest practical boundary.

• Tip 4 – Poisoning questions. The primary defenses are controlling and vetting data sources, access control over the training set, anomaly and outlier detection, label QA, data versioning with hashes, and robust training techniques. Adversarial training is for evasion attacks (inference-time), not poisoning – do not confuse them.

• Tip 5 – Drift is a quality and validation issue in operations. If a model that performed well begins degrading with no code change, the best first step is to compare current input data distributions with training distributions (drift detection), not immediately retrain.

• Tip 6 – Provenance and lineage are the answer to who, where, when questions. If the scenario asks how to determine the origin of a bad record or demonstrate to a regulator what data trained the model, choose lineage tracking, data versioning and dataset documentation.

• Tip 7 – Roles. Data owner is accountable and sets requirements; data steward manages quality day to day; data custodian implements technical safeguards. The AI security manager coordinates but does not own the business data.

• Tip 8 – Bias is a data quality problem first. If asked how to prevent discriminatory outcomes, representative and well-documented training data with bias analysis before training beats post-hoc output adjustment.

• Tip 9 – Feedback loops and generative AI. If outputs are used for retraining, the control is to validate and curate outputs before they re-enter the training set. Watch for the term model collapse.

• Tip 10 – Read the qualifier. BEST, MOST, FIRST and PRIMARY change the answer. FIRST often means assess or define requirements; BEST often means the control that most directly addresses root cause; PRIMARY purpose questions want the main objective, not a side benefit.

• Tip 11 – Documentation is a control. Datasheets, data cards and model cards are legitimate answers for transparency, regulatory compliance and auditability questions.

• Tip 12 – Metrics answers. When asked how to monitor the effectiveness of data controls, choose measurable KRIs such as validation failure rate, duplicate rate, drift index, or percent of datasets with documented provenance – not vague statements like regular review.

8. Practice Scenarios with Reasoning

Scenario A: An organization sources a large image dataset from a public repository to train a classifier. The security manager is MOST concerned about which risk?
Answer approach: Public datasets carry supply-chain risk of data poisoning or backdoor insertion. The best control is source vetting plus integrity verification (hashes/signatures) and anomaly scanning before training. Encryption at rest would be a distractor – it protects confidentiality, not the integrity of what was downloaded.

Scenario B: A credit-scoring model begins rejecting a higher proportion of applicants after six months in production. No code or model change occurred. What should be done FIRST?
Answer approach: Investigate data drift by comparing live input distributions to the training baseline. Retraining without understanding the cause could bake in the problem.

Scenario C: Auditors find that training data was modified by a data engineer after approval without any record of the change. Which control would BEST have prevented this?
Answer approach: Access control with segregation of duties, immutable or versioned storage of approved datasets and hashing of the approved snapshot. Data profiling would not prevent the unauthorized change.

Scenario D: A labeling vendor returns a dataset and the model trained on it shows unexpectedly poor performance on one class. What is the MOST likely cause and control?
Answer approach: Label quality problems (possibly label flipping). Controls: inter-annotator agreement measurement, gold-standard samples, statistical review of label distributions before acceptance.

Scenario E: A regulator asks the organization to demonstrate which data was used to train a high-risk AI system. What capability is required?
Answer approach: Data lineage, dataset versioning and documentation such as datasheets and model cards linking the deployed model to an exact, hashed training snapshot.

9. Quick Revision Summary

• Data quality = fit for purpose; remember the dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness, relevance, representativeness.
• Data integrity = no unauthorized or accidental change; controls include hashing, signatures, immutable storage, access control, segregation of duties, provenance and lineage.
• Validation = checks at every boundary: ingestion, pre-training, inference input, output and feedback loops.
• Key AI-specific threats: poisoning, label flipping, supply-chain tampering, drift, leakage, feedback contamination, bias.
• Governance roles: owner (accountable), steward (quality), custodian (technical).
• Frameworks: NIST AI RMF, ISO/IEC 42001, ISO/IEC 5259, ISO 8000, COBIT APO14, EU AI Act Article 10, OWASP LLM Top 10, MITRE ATLAS.
• Exam mindset: identify quality vs integrity vs validation, locate the lifecycle stage, prefer preventive and root-cause controls, think like a risk-based manager, and read the qualifier carefully.

Mastering this topic positions you to answer not only direct questions on data controls but also many adjacent AAISM questions on model risk, bias, monitoring, supply-chain security and regulatory compliance, because nearly all of them trace back to the quality and integrity of the data.