Chain of Custody for Training and Test Data

Overview
In the PMI Certified Professional in Managing AI (PMI-CPMAI) framework, Chain of Custody for Training and Test Data belongs to the domain of supporting responsible and trustworthy AI. It is the documented, verifiable record of where data came from, who handled it, what was done to it, where it was stored, and how it moved through the AI lifecycle. The lifecycle runs from acquisition through preparation, labeling, splitting, training, testing, deployment and eventual retention or disposal.

The term comes from legal and forensic practice, where evidence is only admissible if every hand-off is recorded and the evidence can be shown to be untampered. Applied to AI, it means a team can prove that the data behind a model is what they claim it is.

Why It Is Important
1. Trust and accountability: A model is only as trustworthy as its data. If you cannot show where data originated or how it was changed, you cannot credibly defend the model's outputs to stakeholders, auditors or regulators.

2. Regulatory and legal compliance: Laws and frameworks such as GDPR, the EU AI Act, HIPAA, CCPA and the NIST AI RMF expect organizations to show lawful basis, consent, data minimization, data governance and traceability. A chain of custody supplies that evidence.

3. Reproducibility: To retrain, debug or validate a model, you must know exactly which data version produced which model version. Without custody records, results cannot be reproduced.

4. Integrity and security: Custody controls help detect and prevent data poisoning, unauthorized changes, accidental corruption and leakage of sensitive information.

5. Preventing data leakage between training and test sets: If test data leaks into training data, performance metrics are inflated and misleading. Custody records prove the separation was maintained.

6. Bias and fairness investigations: When a model behaves unfairly, teams must trace back to the data sources, sampling decisions and labeling processes. Lineage makes root-cause analysis possible.

7. Intellectual property and licensing: Organizations must prove they had the right to use the data. Scraped, licensed or third-party data carries usage restrictions that must be tracked.

8. Incident response and auditability: When something goes wrong, auditors ask 'who touched this data and when?' Chain of custody answers that question.

What It Is: Key Components
Data provenance: The origin of the data, including source system, vendor, collection method, date, consent and license terms.
Data lineage: The path and transformations the data undergoes, such as cleaning, normalization, feature engineering, augmentation, anonymization and labeling.
Custodianship and ownership: Named roles (data owner, data steward, data custodian) responsible at each stage.
Access logs: Records of who accessed, modified, copied or exported data, and when.
Versioning: Unique identifiers or snapshots for every dataset version, linked to the model versions trained or tested on them.
Integrity verification: Checksums, hashes or digital signatures that prove data has not been altered.
Storage and transfer records: Where data resides, how it is encrypted, and how it moves between environments or parties.
Retention and disposal: Policies and evidence of how long data is kept and how it is securely deleted.
Documentation artifacts: Datasheets for datasets, data cards, model cards, data processing agreements and consent records.

How It Works in Practice
Step 1 – Acquisition: Record the source, legal basis, consent, license, collection date, and the person or system that acquired the data. Compute an initial hash or fingerprint.

Step 2 – Ingestion and storage: Place data in a governed repository with access controls such as role-based access and least privilege. Apply encryption at rest and in transit, and turn on logging.

Step 3 – Preparation and transformation: Version every transformation. Use tools such as DVC, MLflow, Delta Lake, Pachyderm, data catalogs or lineage platforms to capture which scripts, parameters and people produced each derived dataset.

Step 4 – Labeling: Track annotators or vendors, labeling guidelines, quality checks and inter-annotator agreement. Third-party labelers must sign agreements covering confidentiality and data handling.

Step 5 – Splitting into training, validation and test sets: Document the split method (random, stratified, temporal or group-based) and the random seeds. Lock and hash the test set. Restrict access to the test set so it cannot influence training.

Step 6 – Training and evaluation: Link each model artifact to the exact dataset versions, code version, hyperparameters and environment. This creates an end-to-end audit trail.

Step 7 – Deployment and monitoring: Continue tracking production data used for monitoring or retraining. Production data feeding back into training must enter the same custody process.

Step 8 – Retention, archiving and disposal: Apply retention schedules. Honor deletion requests such as the right to be forgotten, including their implications for models trained on that data. Document secure disposal.

Governance overlay: All steps are supported by policies, a data governance council or steward, periodic audits, and integration with the organization's risk management and AI ethics processes.

Relation to the CPMAI Methodology
CPMAI follows six iterative phases:

• Business Understanding
• Data Understanding
• Data Preparation
• Model Development
• Model Evaluation
• Model Operationalization

Chain of custody is established primarily in Data Understanding and Data Preparation, and is verified in Model Evaluation. It is then maintained in Operationalization. It is a cross-cutting responsible-AI practice that supports transparency, explainability, privacy, security and accountability.

Common Pitfalls

• Treating data as a one-time asset instead of a versioned, governed artifact.
• Allowing data scientists to copy datasets onto laptops or uncontrolled cloud buckets.
• Reusing the test set repeatedly for tuning, which effectively turns it into training data.
• Failing to record consent and license terms for third-party or scraped data.
• Not linking model versions to data versions.
• Ignoring custody once the model is in production.

Exam Tips: Answering Questions on Chain of Custody for Training and Test Data
Tip 1 – Look for the traceability answer. When a question asks how to prove, defend, audit or reproduce a model's results, the best answer usually involves documented provenance, lineage, versioning and access logs.

Tip 2 – Prefer preventive, process-based controls over reactive fixes. PMI favors establishing governance early, in Data Understanding and Preparation, rather than scrambling after an audit or incident.

Tip 3 – Protect the test set. If a scenario describes inflated accuracy, test data reused for tuning, or overlap between training and test data, the correct response is to restore separation. That means sequestering and locking the test set, documenting the split, and re-evaluating on a clean holdout.

Tip 4 – Know the vocabulary.

• Provenance means origin.
• Lineage means transformations and flow.
• Custodian means the person or team handling the data.
• Integrity means unaltered, usually proven with hashing.
• Versioning means snapshots linked to models.

Distractor answers often swap these terms.

Tip 5 – Connect to compliance and ethics. Questions mentioning GDPR, consent, licensing, sensitive data or regulators point toward chain-of-custody documentation, data processing agreements and access controls.

Tip 6 – Third parties are still your responsibility. If data comes from vendors or labeling contractors, the project manager must ensure contractual and procedural custody controls. Assuming the vendor handles it is usually wrong.

Tip 7 – Choose the answer that involves the right roles. Good answers name or involve data owners, data stewards, legal or compliance staff, and security teams. Weak answers leave it solely to the data scientist.

Tip 8 – Beware of 'fast but untracked' options. Answers that skip documentation to meet deadlines, such as 'retrain quickly with the newest data,' are typically wrong in a responsible-AI context.

Tip 9 – Data poisoning or tampering scenarios. The best answers involve verifying integrity with hashes or checksums, reviewing access logs, rolling back to a known-good dataset version, and tightening access controls.

Tip 10 – Lifecycle thinking. Remember that custody extends to production data, retraining data and disposal. If an option addresses only the training phase while the scenario is about operations, look for a more complete answer.

Sample Question
An AI model shows excellent test accuracy but performs poorly in production. An audit reveals the team cannot identify which dataset version was used for testing. What should the project manager do first?

A) Retrain the model with more data
B) Implement data versioning and lineage tracking, then re-evaluate on a documented, locked holdout set
C) Replace the model algorithm
D) Increase monitoring thresholds

Correct answer: B. It addresses the root cause, which is the missing chain of custody and possible leakage, and restores trustworthy evaluation.

Summary
Chain of custody for training and test data is the governance backbone of trustworthy AI. It ensures data is lawful, traceable, unaltered, properly separated and accountable throughout the AI lifecycle. On the exam, favor answers that emphasize documented provenance and lineage, versioning, access control, integrity verification, test-set protection, clear roles and lifecycle-wide governance.