Model Versioning, Change Management and Rollback

1. Why This Topic Matters
AI models are not static software. They are built from code, training data, hyperparameters, feature pipelines and runtime environments, and any of these can change the model's behaviour. Without disciplined versioning, change management and rollback capability, an organisation cannot answer basic assurance questions such as: Which model made this decision? Who approved its deployment? What changed between the version that worked and the version that failed? How quickly can we return to a known-good state?

For the AAISM exam, this topic sits at the intersection of operational security, integrity assurance, governance and resilience. It links to model poisoning, supply chain risk, drift, incident response, auditability and regulatory accountability (e.g., EU AI Act record-keeping, NIST AI RMF 'Manage' function, ISO/IEC 42001 operational controls). ISACA expects you to treat uncontrolled model change as a risk to confidentiality, integrity, availability, fairness and accountability, and to recognise that the primary remedy is a governed lifecycle, not just technical tooling.

2. What It Is – Core Definitions

Model Versioning
Model versioning is the practice of uniquely identifying and preserving every distinct iteration of a model together with everything required to reproduce and explain it. A complete model version typically captures:
• Model artifacts (weights, parameters, serialised files, container images)
• Training and inference code (commit hash)
• Training dataset version and data lineage (what data, from where, with what preprocessing)
• Hyperparameters and configuration
• Feature definitions and feature store version
• Dependencies and environment (library versions, hardware, random seeds)
• Evaluation results (accuracy, fairness, robustness, safety test outcomes)
• Metadata: owner, purpose, approval status, risk tier, model card or documentation

Versions are usually stored in a model registry, which acts as the authoritative inventory. Many organisations adopt semantic versioning (major.minor.patch) adapted for ML: a major change might be a new architecture or training dataset, a minor change a retraining on refreshed data, and a patch a configuration or threshold adjustment. The exact scheme matters less than consistency, uniqueness and immutability – once a version is registered it should never be silently overwritten.

Change Management
Change management is the governed process by which any modification to a production AI system is requested, assessed, approved, tested, deployed and reviewed. In AI contexts, 'change' is broader than traditional IT because it includes:
• Retraining or fine-tuning a model
• Changing training data sources or data pipelines
• Updating prompts, system instructions, guardrails or RAG knowledge bases (for generative AI)
• Adjusting decision thresholds or post-processing logic
• Swapping a third-party foundation model or API version
• Changing infrastructure, libraries or dependencies that affect model output

A mature AI change management process includes: a documented change request; risk and impact assessment (performance, security, fairness, privacy, regulatory); classification of the change (standard, normal, emergency); testing in a non-production environment; independent approval appropriate to the risk tier; a documented deployment and back-out plan; and post-implementation review. Separation of duties is essential – the person who builds or trains the model should not be the sole approver of its release.

Rollback
Rollback is the ability to revert a production AI system to a previous known-good version quickly, safely and verifiably. Effective rollback depends on versioning (there must be a trusted previous version to return to) and on change management (the rollback itself must be controlled and recorded). Rollback is a corrective control that supports resilience and incident response. It may be triggered by degraded performance, detected bias, security compromise (e.g., discovered poisoning or tampering), safety incidents, regulatory concerns, or failed canary metrics.

3. How It Works in Practice

a) Establishing the Model Registry and Lineage
Every candidate model is registered with a unique identifier and a full lineage record linking data version, code version and experiment run. Artifacts are stored in access-controlled, immutable storage with cryptographic hashes or digital signatures so that integrity can be verified at deployment time. Lifecycle states (e.g., development, staging, production, archived, retired) are tracked in the registry, and promotion between states requires an approval gate.

b) The Change Lifecycle
1. Initiate: A change request documents what is changing, why, the expected benefit, and affected stakeholders.
2. Assess: Impact analysis covers security (new attack surface, data exposure), performance, fairness and bias, explainability, privacy, legal and contractual obligations, and dependencies on downstream systems.
3. Test and validate: The new version is evaluated against the baseline on holdout data, adversarial robustness tests, bias metrics, safety and red-team tests, and integration tests. Results are attached to the version record.
4. Approve: An authority proportional to the model's risk tier approves (for high-risk models this may be a change advisory board, AI governance committee or model risk function). Emergency changes follow an expedited path but still require retrospective approval and documentation.
5. Deploy with a controlled strategy: Blue/green (two parallel environments, switch traffic), canary (small percentage of traffic to the new model, expand if metrics hold), shadow mode (new model runs alongside but its outputs are not used), or A/B testing. These reduce blast radius and make rollback trivial because the previous version is still running.
6. Monitor: Post-deployment monitoring compares the new version against predefined acceptance thresholds (accuracy, latency, drift, error rates, fairness, abuse signals).
7. Review and close: A post-implementation review confirms objectives were met and records lessons learned.

c) Rollback Mechanics
• Predefined triggers: Objective thresholds that automatically or semi-automatically initiate rollback (e.g., accuracy falls below X, fairness disparity exceeds Y, anomalous output rate spikes, integrity check fails).
• Back-out plan: Every change request must include tested rollback steps before approval. A rollback plan that has never been exercised is an assumption, not a control.
• Artifact availability: Previous versions, their dependencies and their compatible data schemas must remain available and verifiable. Rollback can fail if upstream data formats or feature definitions have changed – therefore data and schema versioning must be aligned with model versioning.
• Rollback as a controlled change: Rollback is itself recorded, approved (or retrospectively approved in an emergency) and communicated to stakeholders. The root cause of the failed change must be analysed so the problem is not simply redeployed later.
• Roll-forward alternative: Sometimes reverting is impossible (the old model has a known vulnerability or the data has irreversibly changed). The organisation must then have a fast, governed path to a patched version.

d) Special Considerations for Generative AI and Third-Party Models
• Prompts, system instructions, guardrail configurations, retrieval indexes and fine-tuning adapters are all version-controlled artifacts.
• When relying on vendor APIs, the provider may update the underlying model without notice. Controls include pinning to a specific model version where possible, contractual notification requirements, regression test suites run against the API, and documented contingency plans if behaviour changes.
• Open-source model weights should be verified by hash and signature, with provenance recorded, to mitigate supply chain tampering.

4. Security and Governance Value
• Integrity: Hashing and signing versions prevents undetected substitution or tampering of model files – a direct defence against model poisoning and backdoor insertion in the deployment pipeline.
• Accountability and audit trail: Every production decision can be traced to a specific version, dataset and approver, satisfying regulators, auditors and incident investigators.
• Least privilege and separation of duties: Promotion rights, approval rights and production deployment rights are segregated, limiting insider risk.
• Resilience: Tested rollback reduces mean time to recover from a bad release or an attack.
• Reproducibility: Being able to rebuild a version exactly enables forensic analysis and defensible explanations of model behaviour.
• Drift management: Versioning allows objective comparison between the deployed model and retrained candidates, supporting informed retraining decisions rather than reactive replacement.

5. Key Controls Checklist
• Centralised model registry with unique IDs, lifecycle states and immutable artifacts
• Linked versioning of data, code, configuration, prompts and environment
• Cryptographic integrity verification (hashes, signatures) at build and deploy
• Formal change request, risk/impact assessment and risk-tiered approval
• Separation of duties between development, approval and deployment
• Mandatory pre-production testing including security, bias and safety evaluation
• Progressive deployment (canary, blue/green, shadow)
• Documented and tested back-out/rollback plan for every change
• Defined rollback triggers and monitoring thresholds
• Emergency change procedure with retrospective review
• Complete, tamper-evident logging of all changes and rollbacks
• Retention policy for retired versions supporting audit and legal hold
• Vendor and open-source model provenance and version pinning controls

6. Common Weaknesses Examiners Like to Test
• Versioning only the model weights but not the data or feature pipeline (so rollback silently produces different behaviour)
• Retraining treated as 'business as usual' and exempted from change control
• Data scientists with direct write access to production
• No baseline metrics, so there is no objective basis to decide whether to roll back
• Rollback plan documented but never tested
• Vendor model updates not monitored
• Emergency changes that bypass documentation permanently rather than temporarily
• Assuming a model registry alone equals governance (tool without process)

7. Exam Tips: Answering Questions on Model Versioning, Change Management and Rollback

Tip 1 – Think like a governance professional, not an engineer. ISACA answers favour process, policy, accountability and risk-based decision making. If one option says 'implement a tool' and another says 'establish a governed process with approval and documentation,' the process answer is usually correct unless the question is explicitly technical.

Tip 2 – Read the qualifier: BEST, MOST, FIRST, PRIMARY. 'FIRST' usually points to assessing risk/impact or establishing a baseline before acting. 'BEST' control for integrity is typically cryptographic verification plus access control. 'PRIMARY' purpose of versioning is traceability and reproducibility; the 'PRIMARY' purpose of change management is to reduce risk from unauthorised or untested changes; the 'PRIMARY' purpose of rollback is rapid recovery to a known-good state.

Tip 3 – Rollback is corrective; change management is preventive; versioning is the enabler of both. Questions may ask you to classify controls. Monitoring with rollback triggers is detective leading to corrective action.

Tip 4 – Everything that influences model output is in scope for change control. If a scenario describes retraining, prompt changes, threshold tweaks, new data sources or a vendor API update being made outside change control, the correct answer identifies this as a control gap.

Tip 5 – Separation of duties is a recurring theme. A scenario where the same individual trains, approves and deploys models has an SoD weakness; the best remediation is independent approval and restricted production access, not more training for that person.

Tip 6 – A rollback plan must be tested and the previous version must be verifiable and compatible. If asked what gives the MOST assurance that rollback will work, choose testing the rollback procedure (ideally during deployment rehearsals) over merely documenting it.

Tip 7 – Progressive deployment answers (canary, blue/green, shadow) are the preferred way to reduce release risk. When a question asks how to minimise impact of a potentially faulty model update to users, these strategies beat 'deploy to all users and monitor.'

Tip 8 – Emergency changes still require control. The correct answer never allows permanent bypass; it allows expedited approval with mandatory retrospective documentation and review.

Tip 9 – For third-party models, think contract plus technical safeguard. Version pinning, regression testing against the API, and contractual notification clauses are the expected answers.

Tip 10 – Link to incident response. After a suspected poisoning or tampering, the FIRST technical action is often to verify artifact integrity against the registry hash and revert to a verified version while preserving evidence; the governance action is to invoke incident response and notify stakeholders.

Tip 11 – Eliminate distractors. Wrong options commonly include: relying solely on the data science team's judgment, retaining only the current version to save storage, approving changes after deployment as the norm, or assuming the vendor is responsible for all model change risk.

8. Practice Scenarios

Scenario A: A bank retrains its credit model monthly. A recent retrain increased approval rates for one demographic group and triggered complaints. The team cannot determine which training dataset was used. What is the MOST significant control deficiency?
Answer: Lack of data lineage and version linkage between model and training data. Without it, the issue cannot be reproduced, explained or reliably rolled back. Other options (e.g., insufficient monitoring) are secondary because even detection would not permit root cause analysis.

Scenario B: An organisation wants to minimise user impact when deploying a new fraud detection model. Which approach is BEST?
Answer: Canary or shadow deployment with predefined rollback thresholds, keeping the prior version live. This limits blast radius and makes rollback immediate.

Scenario C: A generative AI vendor silently updates its model and the organisation's chatbot begins producing non-compliant outputs. What should the security manager recommend FIRST?
Answer: Invoke the contingency plan – pin to or revert to the previously validated model version if available and apply compensating guardrails – then address contractual notification and regression testing to prevent recurrence. (If the question asks for the most effective long-term control, choose contractual change notification combined with automated regression testing of the API.)

Scenario D: During an audit, it is found that an ML engineer deployed a hotfix to a production model during an outage without a change ticket. What is the appropriate finding?
Answer: The emergency change procedure was not followed or does not exist; emergency changes must be logged, retrospectively approved and reviewed. The finding is a process gap, not a recommendation to prohibit emergency changes.

Scenario E: Which provides the GREATEST assurance that a model file deployed to production is the one approved by the governance committee?
Answer: Cryptographic signature or hash verification of the artifact against the model registry record at deployment time, enforced by the pipeline.

9. One-Paragraph Summary for Revision
Model versioning gives every model iteration a unique, immutable, fully documented identity (artifacts, data, code, configuration, environment, evaluation and approvals) stored in a registry. Change management ensures every modification affecting model behaviour – including retraining, data, prompts, thresholds and vendor updates – is assessed, tested, independently approved, deployed progressively and reviewed, with separation of duties and an emergency path that is controlled rather than bypassed. Rollback is the tested, governed ability to return to a verified known-good version when predefined triggers fire. Together they deliver integrity, traceability, accountability and resilience, and in the AAISM exam the correct answers consistently favour risk-based governance, process before tooling, testing before trust, and traceability as the foundation of AI assurance.