Training Data Versioning and Experiment Tracking

Overview
Training Data Versioning and Experiment Tracking are core practices within the PMI-CPMAI domain of Managing AI Model Development and Evaluation. They keep AI work reproducible, auditable, governable and manageable. In traditional software, versioning the source code is usually enough. In AI, the behavior of a model depends on three things: the code, the data, and the configuration (hyperparameters, environment and random seeds). If you cannot recover exactly which data and settings produced a given model, you cannot reliably reproduce it, explain it, audit it or improve it.

Why It Is Important
1. Reproducibility: Stakeholders, auditors and regulators may ask how a model was built. Versioned data and tracked experiments let a team rebuild the exact model artifact.
2. Accountability and governance: Trustworthy AI calls for transparency and traceability. Lineage records show which data went into which model, who approved it and when. This supports compliance with regulations such as the EU AI Act, GDPR and sector-specific rules.
3. Root cause analysis: When a deployed model degrades or behaves unfairly, teams need to trace the problem back. It might be a data change, a labeling error, a feature engineering step or a hyperparameter change.
4. Efficient iteration: AI development is iterative, as in CPMAI's iterative phases. Tracking stops teams from repeating failed experiments and makes it fair to compare candidate models.
5. Collaboration: Data scientists, data engineers, MLOps engineers and project managers share one source of truth for what was tried and what worked.
6. Risk management: Rollback is possible when a new model version underperforms. Without a versioned prior state, rollback is guesswork.
7. Data privacy and consent: If a data subject withdraws consent or data must be deleted, versioning shows which datasets and models were affected.

What It Is
Training Data Versioning is the practice of capturing immutable snapshots, or uniquely identifiable versions, of the datasets used to train, validate and test models. Each version records:
- A unique identifier, such as a hash, tag or version number
- The data source and extraction time
- The preprocessing, cleansing and transformation steps applied
- The labeling or annotation version, labeler guidelines and quality metrics
- The train/validation/test split definitions
- Metadata such as schema, statistics, size, owner and data sensitivity classification

Experiment Tracking is the systematic logging of every model training run, or experiment. It typically captures:
- Code version (for example, a Git commit)
- Data version used
- Hyperparameters and configuration
- Algorithm or model architecture
- Environment details (libraries, hardware, random seeds)
- Evaluation metrics (accuracy, precision, recall, F1, AUC, RMSE, fairness metrics, latency)
- Output artifacts (model files, plots, confusion matrices)
- Notes, owner, timestamps and outcome or decision

Together these create lineage, or provenance: a traceable chain from raw data to deployed model. A model registry often sits on top of this. It stores approved model versions with their lineage and lifecycle stage, such as staging, production or archived.

How It Works
Step 1: Establish standards early. During project planning and the data understanding and preparation phases, define naming conventions, versioning policies, metadata requirements, retention rules and tooling. This belongs in the AI project's data management and configuration management plans.
Step 2: Version raw data at ingestion. Snapshot or hash the source data so the original state is preserved. Never overwrite the source data.
Step 3: Version transformations. Treat preprocessing and feature engineering pipelines as code, store them under version control, and link each processed dataset to the pipeline version that produced it.
Step 4: Version labels. Annotation sets change as labels are corrected. Track the label versions and the guidelines used for each.
Step 5: Fix data splits. Record which records went into the training, validation and test sets. This prevents data leakage and ensures fair comparison between models.
Step 6: Log every experiment. Use an experiment tracking tool such as MLflow, Weights and Biases, Neptune, Comet or DVC experiments. Each run links its code version, data version, parameters, metrics and artifacts automatically.
Step 7: Compare and select. Use dashboards to compare runs against the business success criteria and technical KPIs defined in the business understanding phase.
Step 8: Register and promote. Register the chosen model in a model registry with full lineage. Then move it through approval gates, the go/no-go decisions, toward operationalization.
Step 9: Monitor and retrain. In production, link monitoring results such as data drift and model drift to model and data versions. When retraining occurs, new data versions and experiments continue the same lineage.

Common tools: Git (code), DVC, lakeFS, Delta Lake time travel, Pachyderm (data), MLflow, W&B, Neptune (experiments), plus model registries in MLflow, SageMaker, Vertex AI and Azure ML. The exam is tool-agnostic. Focus on the concepts rather than specific products.

Relationship to the CPMAI Methodology
- Phase I, Business Understanding: defines the success metrics that experiments are measured against.
- Phase II, Data Understanding, and Phase III, Data Preparation: this is where data versioning begins.
- Phase IV, Model Development: experiment tracking is central here.
- Phase V, Model Evaluation: tracked metrics support objective comparison and go/no-go decisions.
- Phase VI, Model Operationalization: the registry and lineage support deployment, monitoring, rollback and retraining.
CPMAI is iterative, so teams may loop back to earlier phases. Versioning is what makes those loops safe and traceable.

Common Pitfalls
- Overwriting datasets in place, which makes the original state unrecoverable
- Tracking experiments manually in spreadsheets, which is error-prone and incomplete
- Not recording random seeds or environment details, which leads to irreproducible results
- Evaluating on a test set that changed between experiments, which makes comparisons invalid
- Data leakage between splits
- Not linking a deployed model back to its training data version
- Ignoring label versioning
- Storing sensitive data versions without access controls or retention policies

Exam Tips: Answering Questions on Training Data Versioning and Experiment Tracking
1. Think reproducibility first. If a scenario says a team cannot recreate results, explain a model or identify why performance changed, the answer usually involves implementing data versioning, experiment tracking or lineage.
2. Remember the trio. Model behavior depends on code, data and configuration. Answers that version only code are incomplete. Prefer options that capture all three.
3. Choose proactive, systematic options. PMI favors establishing processes early, in planning and data preparation, over ad hoc fixes after problems appear. Automated logging beats manual spreadsheets.
4. Link to governance and trustworthy AI. When a question mentions auditors, regulators, transparency, explainability or accountability, look for answers about lineage, provenance, documentation and model registries.
5. Drift and degradation scenarios. If a production model degrades, the best first step is often to compare the current data with the versioned training data and review experiment history. Do not jump straight to retraining blindly.
6. Rollback scenarios. If a newly deployed model fails, choose the option that uses the model registry and versioned artifacts to roll back to the last approved version.
7. Fair comparison. When comparing candidate models, the correct answer ensures the same data version, the same test split and the same evaluation metrics, aligned to business success criteria.
8. Watch for data leakage. Fixed, versioned splits prevent test data from contaminating training. Answers that preserve split integrity are typically correct.
9. Privacy and consent. If data must be removed, versioning and lineage identify the affected datasets and models that may need retraining.
10. Iteration is expected. CPMAI is iterative. The right answer usually supports going back to earlier phases with full traceability, rather than starting over or discarding history.
11. Eliminate distractors. Be wary of options that skip documentation to save time, rely on individual memory, overwrite data, or select a model based only on a single metric without business alignment.
12. Project manager's role. The PM does not necessarily run the tools. The PM ensures that standards, roles, tooling and governance for versioning and tracking are defined, resourced and followed, and that they are built into plans and quality gates.

Sample Question Pattern
A model performed well in testing but a stakeholder asks why the newest version produces different predictions than last month's. The team cannot determine the cause. What should the project manager have ensured?
Best answer: that training data versions, code versions and experiment configurations were tracked and linked through lineage, so that the differences could be traced.

Key Takeaway
Training data versioning and experiment tracking turn AI development from an untraceable craft into a disciplined, auditable and repeatable process. On the exam, favor answers that build in traceability, reproducibility and governance early and systematically across the AI lifecycle.