Disaster Recovery for Models, Data and Pipelines

1. Why This Topic Matters

Traditional disaster recovery (DR) was designed around servers, databases and applications. Artificial intelligence changes the picture. An AI system is not a single piece of software; it is a living combination of trained model artifacts, training and inference data, feature engineering logic, data and ML pipelines, configuration and prompts, and specialised infrastructure such as GPUs and vector databases. If any one of these is lost or corrupted, the organisation may be unable to restore the AI system to a trustworthy state, even if the application code is perfectly backed up.

For the ISACA Advanced in AI Risk (AAIR) candidate, this topic sits inside the AI Risk Program Management domain. It tests whether you understand that resilience of AI systems is an enterprise risk issue, not merely an IT task. Regulators, customers and boards increasingly expect that AI systems supporting critical decisions (credit, healthcare, safety, fraud detection, customer service) can be recovered within defined tolerances, that recovered models behave the same as the originals, and that the organisation can prove it through testing and documentation. Frameworks such as the NIST AI Risk Management Framework (Manage function), ISO/IEC 42001 (AI management systems), ISO 22301 (business continuity) and ISO/IEC 27001 (information security, including backup and redundancy controls) all expect continuity and recovery to be planned, tested and governed.

In short: an organisation that cannot recover its models, data and pipelines cannot credibly claim its AI risk is managed.

2. What It Is: Key Definitions

Disaster Recovery (DR) is the set of policies, procedures, tools and capabilities that enable the restoration of technology services, data and systems after a disruptive event (hardware failure, cyber-attack, ransomware, cloud region outage, human error, data corruption, vendor failure, natural disaster).

Business Continuity (BC) is the broader discipline of keeping critical business functions running during disruption. DR is the technology-focused subset of BC. The exam often tests whether you can distinguish the two: BC asks how does the business keep operating; DR asks how do we restore the technology.

Business Impact Analysis (BIA) identifies critical processes, the AI systems that support them, dependencies, and the impact of downtime over time. The BIA is the foundation for setting recovery objectives. On the exam, the BIA almost always comes first before selecting DR strategies or technologies.

Recovery Time Objective (RTO) is the maximum acceptable time to restore a service after disruption. Recovery Point Objective (RPO) is the maximum acceptable amount of data loss measured in time (for example, the last 15 minutes of inference logs or the last daily snapshot of a feature store). Maximum Tolerable Downtime (MTD) is the business-level ceiling that RTO must sit within.

AI-specific recovery assets include:
- Model artifacts: serialised weights, architecture definitions, tokenisers, model cards and signed hashes.
- Training data and labels: the exact versions used to produce a given model, often large and hard to reconstruct.
- Feature stores: curated, versioned features shared between training and inference.
- Pipelines: ETL/ELT jobs, data validation steps, training orchestration (DAGs), CI/CD for ML (MLOps), deployment manifests and infrastructure-as-code.
- Configuration and prompts: hyperparameters, environment definitions, system prompts, retrieval configurations, guardrail rules and policy files.
- Model registry and metadata: lineage, approvals, evaluation results and version history.
- Specialised infrastructure: GPU/TPU capacity, vector databases, embedding stores, inference serving platforms and API keys to third-party foundation models.

Reproducibility is the ability to regenerate an identical or functionally equivalent model from preserved data, code, configuration and environment. It is the AI analogue of restoring a database from backup and is central to DR for models.

Degraded or fallback mode is a pre-approved alternative when the AI system cannot be recovered within RTO: a simpler rules-based engine, a previous stable model version, manual processing, or a safe default decision. Fallback design is a distinctive feature of AI DR because an unavailable model may be safer than a wrong one.

3. How It Works: The AI Disaster Recovery Lifecycle

Step 1: Inventory and classify AI assets. You cannot recover what you do not know you have. A complete AI inventory (models, datasets, pipelines, prompts, vendors, infrastructure) is the starting point. Each asset is assigned a criticality tier based on the BIA. A fraud-scoring model used in real time will have a far tighter RTO than a quarterly marketing propensity model.

Step 2: Define recovery objectives per asset. Set RTO, RPO and MTD for each tier. Recognise that AI adds a third dimension: recovery fidelity, meaning how close the restored model must be to the original. Some organisations define a Recovery Performance Objective, the minimum acceptable accuracy, fairness or latency the recovered system must demonstrate before it is returned to production.

Step 3: Protect model artifacts. Store every production and prior approved model version in a model registry with immutable, versioned, geographically replicated storage. Record cryptographic hashes so that restored artifacts can be verified for integrity (important after ransomware or tampering). Keep at least one known-good previous version readily deployable for rapid rollback.

Step 4: Protect data. Apply backup and replication to training datasets, labels, validation sets, feature stores, embeddings and vector indexes. Use data versioning so a specific model can be linked to the exact dataset snapshot that produced it. Address retention and privacy: backups of personal data must comply with the same data protection controls as production, and deletion requests must propagate to backups within the agreed regime. Follow the 3-2-1 principle (three copies, two media, one off-site or immutable) and protect against ransomware with immutable or air-gapped copies.

Step 5: Protect pipelines and environment. Keep all pipeline code, DAG definitions, infrastructure-as-code, container images, dependency manifests and environment specifications in version control that is itself backed up. Capture library versions and random seeds where determinism matters. Store secrets and API keys for third-party model providers in a recoverable vault with documented rotation procedures. Without the pipeline, the organisation may have the data and the model but be unable to retrain or redeploy.

Step 6: Plan infrastructure recovery. Decide on hot, warm or cold standby for inference serving. Consider GPU scarcity: a cold DR site may be unable to acquire accelerators quickly, so contracts, reserved capacity or multi-region agreements may be required. Evaluate multi-cloud or multi-region deployment for critical models. For third-party foundation models, plan for provider outage, deprecation of a model version, or sudden pricing or terms changes by maintaining an alternate provider, a locally hosted fallback or a non-AI workaround.

Step 7: Design fallback and safe failure. Define in advance what happens if the model is unavailable: fail open, fail closed, route to humans, use the last stable version, or use a transparent rules engine. The choice is a risk decision owned by the business and approved through governance, not improvised by engineers during an incident.

Step 8: Document runbooks and assign roles. Runbooks should state who declares a disaster, who authorises failover, the exact restoration order (infrastructure, data, pipelines, model, validation, cutover), communication steps, and criteria for returning to normal. Roles must span IT operations, data engineering, ML engineering, model risk management, security, legal/privacy and business owners.

Step 9: Validate before restoring service. A restored model must pass integrity checks (hash verification), functional checks (it loads and serves), and performance checks (accuracy, bias and drift tests against a holdout set) before being declared recovered. This is a key difference from traditional DR: a database that restores is done; a model that restores must also be proven to behave correctly.

Step 10: Test, exercise and improve. Conduct tabletop exercises, component restores, full failover tests and chaos experiments on a defined schedule and after significant changes (new model version, new vendor, architecture change). Measure actual recovery time against RTO and actual data loss against RPO. Feed lessons learned into the risk register and update the plan. Untested DR plans are treated by auditors and examiners as unproven.

Step 11: Integrate with governance. DR status, test results and gaps are reported to the AI governance committee or risk committee. DR requirements are embedded into the AI development lifecycle (secure-by-design and resilient-by-design), vendor contracts (SLAs, exit and data return clauses), and the enterprise BC programme.

4. Distinctive AI Risks That Shape DR Planning

Irreproducibility. If training data, random seeds or library versions were not captured, retraining may produce a different model with different behaviour, creating silent model risk after recovery.

Drift during outage. If a model is offline for a long period, the world moves on. The recovered model may be stale, so DR plans should include a re-validation and possibly a retraining step.

Data poisoning and integrity attacks. A disaster may be a corruption event rather than an outage. Recovering from the most recent backup may restore poisoned data. Plans need multiple restore points and integrity verification of training data lineage.

Dependency concentration. Reliance on a single cloud region, GPU vendor or foundation model API creates concentration risk that DR must explicitly address.

Privacy and legal constraints. Replicating training data across regions may breach data residency rules; backups may contain personal data subject to deletion rights. DR design must satisfy privacy and legal requirements, not just availability.

Shadow AI. Models built outside governance will not be in the inventory and therefore not in the DR plan. Inventory completeness is a control.

5. Key Controls and Good Practices Summarised

- Complete AI asset inventory with criticality tiers.
- BIA-driven RTO, RPO and recovery performance objectives.
- Versioned, immutable, replicated model registry with integrity hashes.
- Data and feature versioning linked to each model version.
- Pipeline code, IaC and environment definitions under backed-up version control.
- Secrets management with recoverable, rotatable credentials.
- Redundant or multi-region inference capacity appropriate to tier; reserved accelerator capacity where needed.
- Vendor contracts with SLAs, continuity, exit and data return provisions; alternate providers for critical third-party models.
- Pre-approved fallback modes and safe-failure behaviour.
- Documented runbooks, defined roles and escalation paths.
- Post-restore validation covering integrity, performance, fairness and drift.
- Regular, risk-based DR testing with metrics reported to governance.
- Continuous improvement loop into the AI risk register.

6. Exam Tips: Answering Questions on Disaster Recovery for Models, Data and Pipelines

Tip 1: Think like a risk manager, not an engineer. AAIR questions are written from the perspective of the risk practitioner who advises and the business owner who decides. When asked what should be done FIRST, the answer is usually to perform or update the BIA, identify critical AI assets, or determine business requirements, not to buy replication technology or configure backups. Technology selection follows requirements.

Tip 2: Recognise the sequencing pattern. The exam favours this logical order: inventory and classify, conduct BIA, set RTO/RPO, select strategy, implement controls, document, test, review. If an answer choice jumps ahead in this order while another choice represents the earlier step, the earlier step is usually correct for a FIRST or BEST question.

Tip 3: Know the difference between RTO and RPO cold. RTO is about time to restore service; RPO is about tolerable data loss. If a scenario describes losing the last several hours of inference logs or feature updates, the issue is RPO. If a scenario describes the model being unavailable for too long, the issue is RTO. Questions often swap these to create distractors.

Tip 4: Remember that recovery of a model is not complete until it is validated. If a question asks what is MOST important after restoring a model from backup, look for the answer that involves verifying integrity and validating performance (accuracy, bias, drift) before returning it to production. Simply confirming the system is online is a distractor.

Tip 5: Do not forget the pipeline and the environment. A classic exam scenario: backups of the model and data exist, but recovery fails. The most likely root cause in the answer choices will be missing pipeline code, infrastructure-as-code, dependency versions or configuration. Choose the answer that addresses reproducibility of the full environment.

Tip 6: Third-party and foundation model questions point to contracts and alternatives. When a scenario involves an external API provider outage or model deprecation, the best risk responses are contractual SLAs and exit clauses, alternate providers, locally hosted fallbacks or documented manual workarounds. Blaming the vendor or accepting the risk without analysis are rarely correct.

Tip 7: Testing beats documentation. If asked how to gain assurance that a DR plan will work, the answer is to test it (tabletop, partial restore, full failover) and compare results to objectives. A plan that is approved but untested provides limited assurance. Watch for answer choices that rely on management attestation or policy existence alone.

Tip 8: Fallback modes are risk decisions. If the question asks who should decide whether the AI system fails open or fails closed, or whether to use a previous model version, the answer involves the business owner and governance, informed by the risk practitioner, not the ML engineer acting alone.

Tip 9: Watch for privacy and legal traps. Replicating training data to another jurisdiction, or retaining backups indefinitely, can create compliance risk. If an answer improves availability but ignores data residency or retention requirements, it is probably a distractor. The best answer balances resilience with compliance.

Tip 10: Decode the qualifier words. MOST important usually points to the control that addresses the highest risk or the root cause. BEST points to the most comprehensive, sustainable option. FIRST points to the earliest logical step. PRIMARY purpose of DR testing is to validate that recovery objectives can be met and to identify gaps, not to satisfy auditors.

Tip 11: Eliminate extreme or absolute answers. Options such as backing up everything in real time, retraining all models immediately, or never using third-party models are typically wrong because they ignore cost-benefit and risk-based prioritisation, which is a core AAIR principle.

Tip 12: Map to frameworks when it helps. If the question references NIST AI RMF, DR falls under the Manage function and relates to Govern for accountability. ISO/IEC 42001 expects documented controls for AI system resilience. ISO 22301 supplies the BIA and continuity structure. Knowing these anchors helps you recognise the intended answer.

7. Worked Example Questions

Example 1. An organisation has implemented nightly backups of its production model weights. During a recovery test, the team restored the weights but could not deploy the model because the serving container image and feature transformation code had been lost. What is the BEST recommendation?
Reasoning: The gap is incomplete coverage of recovery assets. The best recommendation is to extend DR scope to include pipeline code, container images, infrastructure-as-code and environment definitions under version control with backups, so the entire system is reproducible. Simply increasing backup frequency of the weights would not solve the problem.

Example 2. A risk practitioner is asked to help establish recovery objectives for a newly deployed real-time credit decisioning model. What should be done FIRST?
Reasoning: Conduct or update the business impact analysis to understand the impact of downtime and data loss on the credit process. RTO and RPO are derived from the BIA, and technology choices come afterward.

Example 3. After a ransomware incident, a model was restored from the most recent backup and immediately returned to production. Later, analysts discover its predictions had degraded. What control was MOST likely missing?
Reasoning: Post-restore validation. The organisation should have verified artifact integrity and tested model performance, fairness and drift against a holdout set before cutover. There is also a possibility that the backup itself contained corrupted or poisoned data, reinforcing the need for multiple restore points and integrity checks.

Example 4. A company relies on a single external foundation model API for its customer support assistant. Which action BEST reduces continuity risk?
Reasoning: Establish a documented fallback: an alternate provider or locally hosted model plus a human or rules-based workaround, supported by contractual SLAs and exit provisions. Accepting the risk without analysis or simply monitoring uptime does not reduce it.

8. One-Page Memory Aid

- DR for AI covers models, data, pipelines, configuration, prompts and infrastructure, not just servers.
- Start with inventory and BIA; derive RTO, RPO and recovery performance objectives.
- Use versioned, immutable, replicated storage for models and data; keep lineage between them.
- Protect pipelines, IaC, environments and secrets so the system is reproducible.
- Plan for GPU scarcity, region failure and third-party provider outage.
- Pre-approve fallback and safe-failure behaviour through governance.
- Validate integrity and performance before declaring recovery.
- Test regularly, measure against objectives, report to governance, and improve.
- On the exam: BIA first, requirements before technology, test over paperwork, validate after restore, business owns the risk decision, balance resilience with privacy and cost.

Master these ideas and you will be able to handle any AAIR question on disaster recovery for models, data and pipelines with confidence.