Disaster Recovery for Models, Data and Pipelines

1. Why Disaster Recovery for AI Matters

Traditional disaster recovery (DR) was built around servers, databases and applications. AI systems add three new classes of assets that can be lost, corrupted or made unavailable: trained models (weights, parameters, configurations), data (training sets, validation sets, feature stores, embeddings, labels, prompts and logs) and pipelines (the automated ingestion, preprocessing, training, evaluation, deployment and monitoring workflows). If any of these is destroyed or compromised, the business process that depends on the AI output stops or, worse, continues producing wrong or unsafe results.

From an AI security management perspective, DR matters because:
• AI increasingly sits in the critical path of revenue, safety and compliance decisions (fraud detection, credit scoring, clinical triage, industrial control).
• Models are expensive and slow to recreate. Retraining a large model may take weeks, specialised hardware (GPUs/TPUs) and data that may no longer be available or lawful to reuse.
• AI assets face unique threats beyond outages: data poisoning, model theft, adversarial manipulation, supply-chain compromise of pretrained models, and silent drift. A DR capability that can roll back to a known-good state is a key security control, not just an availability control.
• Regulators (EU AI Act, DORA, sector rules on operational resilience) expect documented resilience, traceability and the ability to restore trustworthy operation.
• Governance frameworks such as the NIST AI RMF (Manage function), ISO/IEC 42001 and ISACA guidance treat resilience and continuity as explicit management responsibilities for AI systems.

In the AAISM body of knowledge, this topic lives under AI Governance and Program Management because DR is a governance outcome: the AI security manager must ensure that AI continuity requirements are defined, owned, funded, integrated into the enterprise business continuity program, tested and reported on.

2. What It Is: Key Concepts and Definitions

Disaster Recovery (DR) is the set of policies, tools and procedures to restore technology services after a disruptive event. Business Continuity (BC) is broader and concerns keeping the business process running (possibly in degraded mode). AI DR is a subset of enterprise DR that specifically addresses model, data and pipeline assets.

Core metrics you must know:
• RTO (Recovery Time Objective) – maximum tolerable time to restore the AI service after disruption. For a real-time fraud model, RTO may be minutes; for a monthly forecasting model, days.
• RPO (Recovery Point Objective) – maximum tolerable data loss measured in time. For AI this applies to feature store data, inference logs, labels and model versions. A near-zero RPO for the feature store may be required if the model relies on fresh features.
• MTD / MTPD (Maximum Tolerable Downtime / Period of Disruption) – the business-level ceiling that RTO must stay within.
• SDO (Service Delivery Objective) – acceptable level of service during recovery (e.g., falling back to a simpler model or rules engine).

AI-specific recoverable assets:
• Model artifacts: weights/checkpoints, architecture definitions, hyperparameters, tokenizers, fine-tuning adapters, quantised variants, serving containers.
• Data assets: raw training data, curated/cleaned datasets, labels and annotations, validation/test sets, feature store contents, vector databases/embeddings, prompt templates, RAG knowledge bases, inference and audit logs.
• Pipeline assets: pipeline code, orchestration definitions (DAGs), infrastructure-as-code, environment specifications (dependency manifests, container images), secrets and credentials, CI/CD configurations, monitoring and alerting rules, model registry metadata and lineage records.

Reproducibility is the ability to recreate the same model from the same data, code, configuration and random seeds. It is the foundation of AI DR: if you cannot reproduce a model, you can only restore it from a saved artifact.

Known-good state is a validated, signed model version with documented lineage that can be redeployed with confidence. Rolling back to a known-good state is the primary recovery action after poisoning or integrity compromise.

3. How It Works: Building and Operating AI DR

Step 1 – Inventory and classify AI assets. Maintain an AI system inventory (model registry plus data catalogue) that records each model, its data dependencies, pipelines, owners, hosting location, third-party components and business criticality. Without an inventory you cannot scope DR.

Step 2 – Business Impact Analysis (BIA) for AI. For each AI system determine the impact of unavailability, data loss and integrity loss, then derive RTO, RPO and acceptable degraded modes. Tier systems (e.g., Tier 0 safety-critical, Tier 1 revenue-critical, Tier 2 analytical). Tiering drives the recovery strategy and spending.

Step 3 – Select recovery strategies for each asset class.
Models: store versioned, immutable, cryptographically hashed/signed model artifacts in a model registry replicated to a geographically separate location; keep at least the current production version and the previous validated version (N and N-1) to enable rollback; document the exact training recipe for reproducibility; for very large or foundation models, DR may rely on vendor redundancy plus a contractual exit/fallback plan.
Data: apply the 3-2-1 backup principle (three copies, two media, one offsite/immutable) to training data, labels and feature stores; use immutable or write-once storage to defend backups against ransomware and poisoning; snapshot vector databases and RAG corpora; retain lineage so that a restored dataset can be verified against its documented provenance; respect privacy and retention rules so that backups do not become unlawful data stores.
Pipelines: keep all pipeline and infrastructure code in version control; use infrastructure-as-code to rebuild training and serving environments on demand; store container images and dependency lockfiles in replicated registries; back up secrets with a secrets manager that itself has DR; ensure orchestration state (scheduler metadata) is recoverable.

Step 4 – Design resilience patterns for serving. Options include active-active multi-region model serving, warm standby in an alternate cloud region or provider, multi-vendor fallback for API-based models (e.g., switching LLM providers), and graceful degradation to a simpler model, cached predictions, or a deterministic rules engine. Human-in-the-loop fallback is a legitimate continuity strategy for lower-volume, high-stakes decisions.

Step 5 – Integrate security-driven recovery triggers. AI DR is invoked not only by outages but by integrity events: confirmed data poisoning, backdoored model detection, unauthorised model modification, compromised pipeline credentials, or severe drift causing unsafe output. Define decision criteria for when to roll back versus retrain, and who has authority (model owner, AI risk committee, CISO) to declare an AI incident and invoke DR.

Step 6 – Document runbooks. Recovery procedures should specify restoration order (secrets and identity first, then data and feature store, then pipelines, then models, then serving endpoints and monitoring), validation tests to prove the restored model performs within tolerance (accuracy, fairness, latency, safety benchmarks), and communication steps to stakeholders and regulators.

Step 7 – Test regularly. Use tabletop exercises, component restore tests (restore a model from backup and verify its hash and metrics), full failover tests, and chaos engineering on pipelines. Testing must confirm that the restored model is the same model (integrity) and performs acceptably (validity), not merely that a file was copied. Record test results, gaps and remediation.

Step 8 – Manage third-party dependencies. Many AI systems depend on external foundation models, data providers, labelling vendors and cloud ML platforms. Review vendor SLAs, their DR capabilities, data portability, model export rights and exit strategies. Concentration risk (single provider, single region, single GPU type) must be assessed and reported.

Step 9 – Govern, measure and improve. Governance responsibilities include an AI continuity policy aligned to enterprise BCM, assigned ownership (model owners accountable, platform teams responsible), DR requirements embedded in the AI lifecycle and change management (no model goes to production without a recovery plan), metrics such as percentage of Tier 1 models with tested recovery, achieved vs target RTO/RPO, backup integrity verification rate, and periodic reporting to the board or risk committee. Lessons learned from incidents and tests feed back into the plan.

4. Common Pitfalls
• Backing up data but not the model, or the model but not the code and environment needed to serve it.
• Assuming the cloud provider's availability covers your DR – it does not cover your own configuration errors, deleted buckets, poisoned data or compromised credentials.
• Restoring a poisoned or drifted model because the backup window did not go back far enough; retention must exceed the realistic detection time for integrity attacks.
• Restoring training data in violation of deletion requests or retention limits.
• Never testing recovery, or testing only file restoration without validating model performance.
• Ignoring GPU/accelerator scarcity in the recovery region, making the RTO unachievable.
• Treating DR as a purely technical task rather than a governance obligation with defined ownership and executive oversight.

5. Exam Tips: Answering Questions on Disaster Recovery for Models, Data and Pipelines

Tip 1 – Think like an AI security manager, not an engineer. AAISM questions favour answers about governance, risk alignment, ownership, policy and assurance over specific tools. If a question asks what the security manager should do first, the answer is usually to perform or update the BIA / risk assessment, establish requirements, or ensure the AI assets are in the inventory – not to buy a backup product.

Tip 2 – Business impact drives RTO and RPO. When asked how recovery objectives should be determined, select the option tied to business criticality and impact analysis, never the one based on technical convenience, budget or what the vendor offers.

Tip 3 – Remember the triad of recoverable assets. If an answer option covers only models, or only data, it is likely incomplete. The best answer typically addresses models, data and pipelines (including code, configuration, environments and secrets) so the system can be rebuilt end to end.

Tip 4 – Integrity is part of recovery. Questions describing poisoning, tampering or unauthorised changes are testing whether you know to roll back to a validated, signed known-good version and verify integrity (hashes, signatures, lineage) before redeploying. Simply restoring the most recent backup is often the distractor because the most recent copy may be compromised.

Tip 5 – Validate, don't just restore. The best answer about DR testing confirms the restored model meets performance, safety and fairness thresholds, not just that files were recovered. Look for words like 'validate', 'verify' and 'test against baseline'.

Tip 6 – Prefer immutable, versioned, offsite and replicated. When choosing backup approaches, options that mention immutability, versioning, separation from production credentials and geographic diversity are stronger than single-location or mutable copies.

Tip 7 – Degraded operation is acceptable. Fallback to a simpler model, cached outputs, a rules engine or human review is a valid continuity strategy. Don't dismiss an answer because it does not restore full AI capability immediately; continuity of the business process is the goal.

Tip 8 – Third-party AI still requires your plan. If the scenario involves an external foundation model or ML platform, the correct answer usually involves reviewing SLAs and DR provisions, ensuring data/model portability, and having a documented fallback or exit strategy – not assuming the vendor handles everything.

Tip 9 – Integrate, don't isolate. AI DR should be integrated into the enterprise business continuity and incident response programs. An answer proposing a separate, standalone AI DR process with no linkage to enterprise BCM is normally wrong.

Tip 10 – Watch qualifier words. 'MOST important', 'BEST', 'FIRST', 'PRIMARY' change the answer. 'First' points to assessment and requirements; 'most important' points to business alignment or executive accountability; 'best' points to the most complete and risk-based control.

Tip 11 – Privacy and compliance constraints still apply to backups. If an option involves retaining all training data indefinitely or copying regulated data to an unapproved region, it is a distractor. Correct answers respect retention, deletion and data residency requirements.

Tip 12 – Eliminate absolutes and tool-centric answers. Options with 'always', 'never', 'all data' or naming a single technology as the complete solution are rarely correct in ISACA-style exams.

6. Sample Question Patterns

Pattern A: 'An organisation discovers that its production fraud model was trained on poisoned data three weeks ago. Which recovery action is MOST appropriate?' → Roll back to the last validated model version trained before the poisoning, verified by integrity checks and performance testing, while quarantining the affected data and pipeline.

Pattern B: 'What should the AI security manager do FIRST when establishing DR for a newly deployed model?' → Ensure the model and its dependencies are inventoried and classified, and conduct a business impact analysis to set RTO/RPO.

Pattern C: 'Which of the following BEST demonstrates the effectiveness of an AI DR plan?' → Documented results of a recovery test showing the restored model met performance and integrity criteria within the defined RTO and RPO.

Pattern D: 'A company relies on a single third-party LLM API for customer service. What is the GREATEST concern?' → Concentration/vendor dependency risk without a tested fallback or exit strategy.

7. One-Line Summary
AI disaster recovery means being able to restore trusted models, lawful and intact data and reproducible pipelines within business-defined RTO/RPO, triggered by both outages and integrity compromises, governed as part of the enterprise continuity program, and proven through regular testing that validates performance – not just file restoration.