Model Retraining, Versioning and Change Control

1. Why This Topic Matters

AI models are not static assets. Unlike traditional software, a model's behavior depends on the data it was trained on, the parameters it learned, the features it consumes and the environment it operates in. Each of these can change after deployment. Real-world data shifts (data drift), the relationship between inputs and outcomes changes (concept drift), business rules evolve, regulations change and vulnerabilities are discovered. If a model is simply left running, its accuracy, fairness and safety degrade silently.

Retraining addresses performance decay, but retraining without discipline introduces new risks: an updated model may behave differently, introduce new bias, break downstream integrations, or quietly expand scope beyond what was approved. Versioning and change control are the governance mechanisms that make retraining safe, traceable and auditable. For the ISACA AAIR exam, this topic sits inside the AI Life Cycle Risk Management domain and tests whether you understand that a model is a managed, changing asset that requires the same rigor as any production system, plus additional AI-specific controls.

Key risk themes examiners expect you to recognize:
- Unmanaged retraining can cause silent regressions in accuracy, fairness or safety.
- Without versioning, you cannot reproduce a decision, investigate an incident or satisfy a regulator asking 'which model made this decision?'
- Without change control, there is no accountability, no segregation of duties and no rollback path.
- Retraining on new data may introduce data poisoning, privacy violations or licensing issues.
- Updates to third-party or foundation models can change behavior outside the organization's control.

2. What It Is – Core Definitions

Model Retraining is the process of updating a model by training it again, either on new data, on an expanded dataset, with new features, or with revised hyperparameters. Retraining may be:
- Scheduled / periodic (e.g., monthly) – predictable, easy to govern.
- Triggered / event-driven – initiated when monitoring detects drift, a performance threshold breach, a fairness metric violation, a security incident or a business/regulatory change.
- Continuous / online learning – the model updates itself as new data arrives. This is the highest-risk mode because changes happen without a human approval step, and the exam will often frame it as requiring compensating controls (guardrails, bounded updates, shadow evaluation, kill switches).

Fine-tuning is a related concept for foundation and large language models: adapting a pre-trained model to a specific task with a smaller dataset. From a governance standpoint it is treated as a retraining event and produces a new version.

Model Versioning is the practice of uniquely identifying and preserving each distinct state of a model so that it can be reproduced, compared, audited and rolled back. True AI versioning covers more than the model file. It includes:
- The training data snapshot (or a hash/reference to it) and its lineage.
- The feature engineering code and feature definitions.
- The training code, libraries, dependencies and environment.
- Hyperparameters and configuration.
- The resulting model artifact (weights/parameters).
- Evaluation results (accuracy, fairness, robustness, explainability metrics).
- Documentation such as model cards and data sheets.
- For generative AI: the prompt templates, system prompts, retrieval indexes and guardrail configurations, which can change behavior as much as the weights.

The principle to remember: same data + same code + same configuration = same model. If any element is not versioned, reproducibility is lost.

Change Control (change management) is the formal process that ensures any modification to a production AI system is requested, assessed, tested, approved, documented, deployed and reviewable. It applies to retraining, but also to changes in thresholds, features, prompts, integrations, data sources, infrastructure and vendor model updates. Change control exists to protect integrity, availability and accountability, and in AI it also protects fairness, safety and compliance.

3. How It Works – The Mechanics

A. Retraining triggers and decision gates
Monitoring feeds the retraining decision. Typical triggers include:
- Performance metrics (accuracy, precision, recall, error rates) falling below defined thresholds.
- Statistical drift detected in input distributions or prediction distributions.
- Fairness or bias metrics breaching tolerances for protected groups.
- New regulatory requirements or policy changes.
- Discovery of data quality problems, label errors or poisoned data.
- Security incidents such as adversarial manipulation or model extraction attempts.
- Business change: new products, markets, customer segments or processes.

Good practice is to define thresholds and triggers before deployment so retraining is a controlled response rather than an ad hoc reaction.

B. The retraining pipeline
1. Data acquisition and validation – new data is collected, checked for quality, completeness, representativeness, consent/legal basis and potential poisoning. Data lineage is recorded.
2. Training in a controlled environment – separate from production, with access controls and logging.
3. Evaluation against a baseline – the candidate model is compared to the current production model on a held-out test set and on fairness, robustness, explainability and safety criteria. Regression testing confirms that previously fixed issues have not returned.
4. Validation and independent review – ideally by someone other than the developer (segregation of duties), with results documented.
5. Approval – by the designated model owner, risk function or AI governance committee, proportionate to the model's risk tier.
6. Controlled deployment – using techniques such as shadow mode (running alongside the old model without affecting outcomes), canary release (small percentage of traffic), A/B testing or blue-green deployment.
7. Post-deployment monitoring – confirming the new version behaves as expected and that rollback is available.

C. Versioning mechanics
- A model registry acts as the system of record: it stores every version, its metadata, lineage, evaluation results, approval status and lifecycle stage (development, staging, production, retired).
- Semantic or structured version numbering communicates the nature of a change (e.g., major change in architecture or data vs. minor hyperparameter tweak).
- Immutability: once a version is registered, it is not modified; a change creates a new version.
- Traceability of decisions: production systems log which model version produced each output, so any individual decision can be traced back to a specific, reproducible model. This is essential for incident investigation, complaint handling, explainability obligations and regulatory audits.
- Retention and retirement: old versions are retained for as long as legal, audit or business needs require, then securely decommissioned.

D. Change control mechanics
- Change request: documents what is changing, why, and who is requesting it.
- Risk and impact assessment: considers effects on performance, fairness, privacy, security, downstream systems, users and compliance. A material change may require a refreshed AI impact assessment or re-approval at a higher governance level.
- Classification of change: standard (pre-approved, low risk, repeatable such as scheduled retraining within defined bounds), normal (requires assessment and approval) and emergency (expedited to address an incident, with retrospective review).
- Testing and validation evidence attached to the request.
- Approval authority defined by risk tier; high-risk models need senior or committee approval.
- Segregation of duties: the person who builds the model should not be the sole person who approves and deploys it.
- Deployment with rollback plan: the previous version remains available and the steps to revert are tested.
- Documentation and communication: model cards, release notes, stakeholder notification, and updates to the AI inventory.
- Audit trail: complete, tamper-evident records of who changed what, when, why and with what approval.

E. Special considerations
- Third-party and foundation models: vendors may update models without notice. Controls include contractual notification clauses, version pinning, re-validation after vendor updates, and monitoring for behavioral change.
- Continuous learning systems: require bounded learning rates, constraints on what can change, periodic human review, anomaly detection on model behavior and the ability to freeze or revert.
- Retraining and privacy: new training data must have a lawful basis; data subject deletion requests may require retraining or machine unlearning considerations.
- Retraining and bias: new data can reintroduce or amplify bias; fairness testing must be repeated on every version, not just the first.
- Documentation drift: model cards and risk assessments must be updated with each material version, or governance documentation becomes misleading.

4. Mapping to Risk Management Principles

The exam frequently frames this topic through classic governance lenses:
- Accountability: a named model owner is responsible for each version and each change.
- Traceability and reproducibility: versioning of data, code, config and artifacts.
- Integrity: change control prevents unauthorized or untested modifications.
- Availability and resilience: rollback capability and staged deployment limit the blast radius of a bad update.
- Transparency: documentation and release notes keep stakeholders informed.
- Proportionality: the rigor of change control should scale with the model's risk classification; a high-risk model affecting individuals' rights demands far stricter gates than an internal recommendation tool.
- Continuous assurance: monitoring, retraining and change control form a loop, not a one-time event.

5. Exam Tips: Answering Questions on Model Retraining, Versioning and Change Control

Tip 1 – Identify which of the three controls the question is really about. Retraining questions focus on when and why to update and the risks of doing so. Versioning questions focus on reproducibility, traceability and rollback. Change control questions focus on authorization, testing, approval, segregation of duties and audit trail. Pick the answer that directly addresses the control gap described.

Tip 2 – 'Cannot reproduce' or 'cannot determine which model made the decision' = versioning. If a scenario says an organization cannot explain a past output, cannot recreate a model, or does not know which version is in production, the best answer involves a model registry, version control of data/code/artifacts, or logging the model version with each prediction.

Tip 3 – 'Deployed without approval' or 'developer pushed directly to production' = change control. Look for answers about formal change management, segregation of duties, independent validation and documented approval. A single person building, approving and deploying is a classic control weakness.

Tip 4 – 'Performance degraded over time' = monitoring that triggers retraining, not retraining alone. The best answer usually involves defining performance and drift thresholds and establishing a governed retraining process. Avoid answers that suggest retraining blindly or more frequently without evaluation.

Tip 5 – Retraining is a change, so it must go through change control. If an option implies retrained models can bypass testing or approval because 'it is the same model,' that option is wrong. Every retrained model is a new version that must be evaluated against the baseline, including fairness and safety metrics, before release.

Tip 6 – Prefer answers that compare the new version to the old one. Baseline comparison, regression testing, champion/challenger evaluation, shadow deployment and canary releases are strong answers because they detect regressions before full exposure.

Tip 7 – Rollback is the resilience answer. When a question asks what should be in place before deploying a new version to minimize impact if it fails, the answer is a tested rollback plan with the prior version retained and available.

Tip 8 – Continuous or online learning questions want compensating controls. Expect answers about bounded updates, human oversight, anomaly detection on behavior, the ability to freeze or revert, and periodic formal validation. Answers that simply say 'disable continuous learning' may be too absolute unless the model is high-risk and the question emphasizes regulatory constraints.

Tip 9 – Third-party model updates are still your risk. If a vendor changes a model, the organization should have contractual notification rights, pin versions where possible, and re-validate before accepting the update. 'Rely on the vendor's testing' is rarely the best answer.

Tip 10 – Documentation must follow the version. Model cards, impact assessments, AI inventory entries and risk registers should be updated with each material change. If a question notes that documentation describes an old version, the gap is documentation and inventory update within the change process.

Tip 11 – Apply proportionality and risk-based thinking. ISACA exams reward answers that scale controls to risk. For a high-risk model, choose the stricter option (committee approval, independent validation, full re-assessment). For a low-risk internal model, a pre-approved standard change path for scheduled retraining may be the best answer.

Tip 12 – Watch for the 'BEST', 'FIRST', 'MOST important' qualifiers. 'FIRST' often points to assessment or defining criteria (thresholds, triggers, approval authority) before action. 'BEST' usually points to the preventive, governance-level control (formal change management, model registry) over a detective or reactive one. 'MOST important' for auditors often points to evidence: audit trails, approval records and version logs.

Tip 13 – Eliminate answers that remove human accountability. Fully automated approval, automatic promotion to production based only on an accuracy score, or skipping fairness testing because accuracy improved are typical distractors.

Tip 14 – Remember the AI-specific additions to traditional change management. Beyond code testing, AI changes require data validation, bias/fairness re-testing, robustness testing, explainability checks and, for generative models, prompt and guardrail versioning. An answer that treats a model like ordinary software and ignores data and fairness is usually incomplete.

6. Quick Worked Example

Scenario: A credit scoring model is retrained quarterly by the data science team, which deploys the new model directly to production after confirming accuracy improved on an internal test set. A regulator later asks why a specific applicant was declined six months ago, and the team cannot determine which model version was used.

What are the control weaknesses? (1) No independent validation or approval – change control gap and lack of segregation of duties. (2) Evaluation limited to accuracy – fairness, stability and regression testing omitted. (3) No linkage between individual decisions and model versions – versioning and logging gap. (4) Likely no model registry or immutable artifact storage – reproducibility gap.

Best remediation answer on an exam: Implement a formal change management process requiring independent validation (including fairness testing) and documented approval before deployment, supported by a model registry that stores immutable versions with lineage and logs the version identifier against every production decision, with a tested rollback capability.

7. One-Line Summary to Memorize

Monitor to know when to retrain; retrain through a governed pipeline; version everything (data, code, config, artifact, docs); control every change with assessment, independent testing, approval, audit trail and rollback; and scale the rigor to the model's risk.