MLOps and Development Environment Controls

Overview
This guide covers MLOps and Development Environment Controls, a topic within the ISACA Advanced in AI Risk (AAIR) domain of AI Life Cycle Risk Management. It explains why these controls matter, what they consist of, how they operate in practice, and how to reason through exam questions on the subject.

1. Why This Topic Is Important
Traditional software is built once and behaves deterministically. AI and machine learning (ML) systems are different: they are built from data, code, and models, all of which change over time. A model that was accurate and fair at release can silently degrade as data shifts, as upstream pipelines change, or as a developer retrains it with a slightly different dataset. Without disciplined controls over the environments and pipelines in which models are developed, tested, deployed, and monitored, an organization cannot answer basic governance questions such as: Which version of the model is in production? What data was it trained on? Who approved the release? Can we reproduce the result? Can we roll back?

From a risk perspective, weak development environment and MLOps controls lead to:
• Unreproducible results – inability to recreate a model for audit, regulatory inquiry, or incident investigation.
• Unauthorized or untested changes – models or data pipelines promoted to production without review, introducing bias, errors, or security flaws.
• Data leakage and privacy breaches – production data copied into loosely controlled development sandboxes.
• Supply chain compromise – poisoned open-source models, libraries, or datasets entering the pipeline.
• Model drift going undetected – performance degradation without monitoring or retraining triggers.
• Regulatory non-compliance – frameworks such as the EU AI Act, NIST AI RMF, and ISO/IEC 42001 expect documented, traceable, and controlled life cycle processes.

For the AAIR exam, ISACA expects candidates to understand that MLOps is the operational control environment for AI, analogous to change management, SDLC controls, and IT general controls (ITGCs) in traditional IT audit—but adapted for the data- and model-centric nature of AI.

2. What It Is
MLOps (Machine Learning Operations) is the set of practices, tools, and governance processes that bring DevOps-style discipline to the ML life cycle: data ingestion and preparation, feature engineering, experimentation, training, validation, deployment, monitoring, and retirement. Its goal is to make ML delivery repeatable, reliable, auditable, and scalable.

Development Environment Controls are the safeguards applied to the environments where data scientists and ML engineers work—notebooks, sandboxes, training clusters, feature stores, model registries, and CI/CD pipelines. These controls ensure that development activities are segregated from production, that access is restricted, that artifacts are versioned, and that nothing reaches production without appropriate testing and approval.

Key components you should be able to define:
• Environment segregation – distinct development, test/staging, and production environments, with controlled promotion between them.
• Version control – for code (e.g., Git), data (dataset versioning/hashing), and models (model registry with version tags).
• Model registry – a central inventory of trained models, their versions, metadata, lineage, approval status, and deployment state.
• Feature store – a governed repository of reusable, versioned features that ensures consistency between training and inference (reduces training-serving skew).
• Experiment tracking – logging of hyperparameters, datasets, metrics, and artifacts for each training run so results can be reproduced.
• CI/CD/CT pipelines – Continuous Integration, Continuous Delivery/Deployment, and Continuous Training automating build, test, validation, and release with embedded quality gates.
• Data and model lineage / provenance – end-to-end traceability of where data came from, how it was transformed, and which model consumed it.
• Access control and identity management – least privilege, role-based access, MFA, and privileged access management for data, compute, and pipelines.
• Secrets management – protection of API keys, credentials, and tokens used in pipelines.
• Dependency and supply chain controls – vetted libraries, software bills of materials (SBOMs/AI-BOMs), scanning of pre-trained models and packages.
• Infrastructure as Code (IaC) – declarative, version-controlled environment configuration to prevent drift and undocumented changes.
• Monitoring and observability – tracking of model performance, data drift, concept drift, latency, and fairness metrics in production with alerting.
• Rollback and incident response – ability to revert to a prior model version quickly when issues are detected.
• Documentation artifacts – model cards, datasheets for datasets, and risk assessments stored alongside the model.

3. How It Works
Think of MLOps as a controlled pipeline with gates at each transition.

Step 1 – Development/Experimentation. Data scientists work in sandboxed environments using de-identified, synthetic, or approved subsets of data. Notebooks and code are committed to version control. Each experiment is tracked (parameters, data version, metrics). Access is role-based; production credentials are not available here.

Step 2 – Data Validation and Preparation. Automated checks validate schema, completeness, distribution, and quality of training data. Data lineage is recorded. Sensitive data is classified and handled according to policy. Feature engineering is performed through the feature store so the same transformations apply in training and inference.

Step 3 – Training and Evaluation. Training jobs run on controlled compute with logged configurations. Models are evaluated against predefined acceptance criteria—accuracy, robustness, fairness/bias metrics, explainability requirements, and security tests (e.g., adversarial robustness). Results are compared to baselines and champion models.

Step 4 – Registration and Approval. Candidate models are registered in the model registry with metadata, lineage, evaluation results, and documentation (model card). A formal review—ideally involving risk, compliance, and business owners—approves the model for staging or production. Segregation of duties ensures the developer is not the sole approver.

Step 5 – Deployment. CI/CD pipelines package and deploy the model through staging, with integration and performance tests. Deployment strategies such as canary releases, blue/green, shadow mode, or A/B testing limit blast radius. Infrastructure is provisioned via IaC. Release records are retained for audit.

Step 6 – Monitoring and Continuous Training. Production monitoring tracks input data drift, prediction drift, performance decay, fairness metrics, latency, and anomalous usage. Thresholds trigger alerts, human review, or automated retraining (Continuous Training). Any retrained model must re-enter the validation and approval gates—automation does not bypass governance.

Step 7 – Retirement and Rollback. Deprecated models are archived with their lineage for regulatory retention. If a production issue occurs, the registry enables rapid rollback to a known-good version.

Mapping to Risk and Assurance. For an auditor or risk professional, these controls map to familiar ITGC categories: change management (approval gates, segregation of duties, testing before release), logical access (RBAC, least privilege, secrets), operations (monitoring, incident response, backup/rollback), and program development (requirements, validation, documentation). The AI-specific additions are data versioning, model registries, drift monitoring, fairness testing, and lineage.

4. Common Risks and Corresponding Controls
• Risk: Developer trains a model on production PII in a notebook. Control: Data classification, masked/synthetic data in dev, DLP, access restrictions.
• Risk: Model promoted to production without testing. Control: CI/CD quality gates, mandatory validation, approval workflow, segregation of duties.
• Risk: Cannot reproduce a model under regulatory inquiry. Control: Versioning of code, data, and models; experiment tracking; lineage records.
• Risk: Training-serving skew causes production errors. Control: Feature store with shared transformation logic.
• Risk: Compromised open-source model or library. Control: Vetted repositories, dependency scanning, AI-BOM, integrity verification (hashes/signatures).
• Risk: Silent model degradation. Control: Drift and performance monitoring with thresholds and alerting; scheduled revalidation.
• Risk: Undocumented infrastructure changes. Control: Infrastructure as Code with peer review and configuration baselines.
• Risk: Automated retraining introduces bias. Control: Retraining must pass the same fairness and validation gates; human-in-the-loop approval for high-risk models.

5. Exam Tips: Answering Questions on MLOps and Development Environment Controls
Tip 1 – Think like a risk professional, not an engineer. AAIR questions rarely test tool names. They test whether you can identify the control objective (traceability, integrity, authorized change, confidentiality, availability) and select the control that best achieves it. When you see a scenario, ask: what could go wrong here, and which option prevents or detects it?

Tip 2 – Map to traditional ITGCs. If a question describes a developer deploying directly to production, that is a change management and segregation of duties failure. If it describes copying production data into a sandbox, that is a data protection / logical access issue. Use your existing audit knowledge and layer AI specifics on top.

Tip 3 – Remember the three things that must be versioned: code, data, and models. Many questions hinge on reproducibility. If an option mentions versioning only code, it is incomplete; the best answer typically includes data and model versioning or lineage.

Tip 4 – Prefer preventive and automated gates over after-the-fact review. When asked for the best or most effective control, options that embed validation, fairness testing, and approval into the pipeline (quality gates) usually outrank manual periodic reviews. However, for high-risk models, a human approval step is expected—automation does not replace accountability.

Tip 5 – Know what a model registry and feature store are for. Model registry = inventory, versioning, approval status, lineage, rollback. Feature store = consistent, reusable, governed features that prevent training-serving skew. Questions often present these as answer options; pick based on the stated problem.

Tip 6 – Monitoring questions: distinguish data drift from concept drift. Data drift = input distribution changes; concept drift = relationship between inputs and target changes. Both require production monitoring with thresholds and a defined response (retrain, recalibrate, or retire). The "best" answer usually includes both detection and a defined action.

Tip 7 – Watch for "first" or "most important" qualifiers. If asked what to do first when establishing controls, the answer is typically to inventory models/establish a registry or define policy and requirements—governance precedes tooling. If asked what is most important for auditability, the answer is lineage/traceability.

Tip 8 – Segregation of duties applies to ML too. The person who develops the model should not be the sole approver of its production release. Options that split development, validation, and deployment roles are usually correct.

Tip 9 – Supply chain awareness. Pre-trained models, datasets, and libraries from external sources are risks. Correct answers involve vetting sources, verifying integrity, maintaining a bill of materials, and scanning for vulnerabilities—not simply trusting popular repositories.

Tip 10 – Eliminate extremes. Options that "prohibit all use of notebooks," "require manual review of every prediction," or "fully automate deployment without human oversight" are typically distractors. ISACA favors balanced, risk-based, proportionate controls.

Tip 11 – Documentation is a control. Model cards, datasheets, and approval records are not bureaucracy—they are evidence. If a question asks how to demonstrate compliance or support an audit, documentation and lineage artifacts are strong answers.

Tip 12 – Read the scenario for the life cycle phase. Identify whether the question concerns development, validation, deployment, or operations. The appropriate control differs by phase: sandboxing and data masking in development; testing and approval in validation; canary/blue-green and IaC in deployment; drift monitoring and rollback in operations.

6. Quick Review Checklist
• Separate dev/test/prod environments with controlled promotion.
• Version code, data, and models; track experiments.
• Central model registry with lineage, metadata, and approval status.
• Feature store to ensure training-serving consistency.
• CI/CD/CT pipelines with automated quality, security, and fairness gates.
• Least-privilege access, MFA, secrets management, no production data in uncontrolled sandboxes.
• Vetted dependencies, AI-BOM, integrity checks for third-party models and data.
• Infrastructure as Code to prevent configuration drift.
• Production monitoring for drift, performance, fairness, and security with defined response actions.
• Rollback capability and incident response procedures.
• Documentation (model cards, datasheets, approvals) retained for audit and regulatory needs.
• Segregation of duties and human oversight for high-risk models.

Summary
MLOps and development environment controls translate the principles of controlled software delivery into the AI context, where data and models are as critical as code. For the AAIR exam, focus on the control objectives—reproducibility, authorized change, integrity, confidentiality, and ongoing performance assurance—and select answers that embed governance into the pipeline while retaining appropriate human accountability. When in doubt, choose the option that provides traceability, enforces a gate before production, and is proportionate to the model's risk level.