Ongoing Maintenance and Support Procedures

Overview
In the PMI Cognitive Project Management for AI (PMI-CPMAI) methodology, the final phase is Operationalize AI Solution (CPMAI Phase VI). Deploying a model is not the end of an AI project. It is the start of its operational life.

Ongoing Maintenance and Support Procedures are the defined, repeatable processes, roles and tools that keep an AI solution accurate, reliable, secure, compliant and valuable after it goes into production.

Traditional software behaves the same way until someone changes its code. AI systems are different. Their behavior depends on data, and real-world data changes constantly. This is why CPMAI stresses that AI projects are iterative and that operationalization must plan for continuous care.

Why It Is Important
1. Models degrade over time (model drift). Accuracy erodes as conditions change:
- Data drift is a shift in the input data distribution.
- Concept drift is a change in the relationship between inputs and outcomes.
Example: a fraud model trained before a new payment method appeared may miss new fraud patterns.

2. Business value must be sustained. The success criteria set in Business Understanding (Phase I) must keep being met in production, not just on deployment day.

3. Trustworthy AI obligations continue. Bias, fairness, transparency, explainability, privacy and regulatory compliance must be monitored on an ongoing basis. New data can introduce new bias.

4. Operational risk. Several things can cause outages, wrong decisions or security incidents:
- broken data pipelines
- schema changes
- infrastructure failures
- adversarial attacks

5. Cost control. Unmonitored compute, storage and retraining costs can grow quietly. Maintenance procedures keep total cost of ownership in check.

6. Accountability. Clear ownership prevents an 'orphaned model' that nobody monitors or can fix.

What It Is: Key Components
Model performance monitoring
- Track the KPIs and model metrics defined earlier, such as accuracy, precision/recall, F1, RMSE, latency and throughput.
- Compare them against baselines and thresholds.

Data monitoring
- Watch data quality: missing values, outliers and schema changes.
- Watch data distribution for drift.
- Watch data pipeline health and data lineage.

Drift detection and alerting
- Use automated statistical checks, such as PSI, KS tests and changes in prediction distribution.
- Alerts fire when a threshold is breached.

Retraining strategy
Define when and how models are retrained:
- scheduled (for example, monthly)
- trigger-based (for example, when drift or a drop in performance is detected)
- continuous / online learning (used carefully)
Include validation gates before redeployment.

Model versioning and governance
- Keep a model registry and version data, code and models.
- Use approval workflows, audit trails and documentation (model cards).

Rollback and contingency plans
- Be able to revert to a previous model version or a rule-based fallback.
- Use human review when the model misbehaves.

Support model and roles
- Define tiered support (L1, L2, L3), escalation paths and service level agreements (SLAs).
- Assign a model owner plus support from data science, data engineering, MLOps/IT and the business.

Incident management
Follow a defined process: detect, triage, resolve, root-cause analysis, then lessons learned.

Feedback loops
- Capture user feedback, ground-truth labels and business outcomes.
- Feed them back into evaluation and retraining.

Security and compliance maintenance
- Manage access control and patching.
- Protect against adversarial attacks and data poisoning.
- Run periodic audits for regulatory and ethical compliance.

Infrastructure and cost management
Handle scaling, resource usage, dependency updates and cloud cost monitoring.

Change management and communication
Keep stakeholders and end users informed of model updates, changes in behavior and known limitations.

Decommissioning criteria
Know when a model should be retired or replaced.

How It Works: The Lifecycle in Practice
Step 1: Plan before deployment. Maintenance and support procedures should be designed during the Model Evaluation and Operationalization planning, not after go-live. This includes:
- monitoring metrics, thresholds and owners
- retraining cadence
- SLAs
- the budget for ongoing operations

Step 2: Establish baselines. Record the model's performance and data profile at deployment so future changes can be measured.

Step 3: Monitor continuously. Automated dashboards and alerts track:
- model metrics
- data quality
- drift
- system health
- fairness metrics
- business KPIs

Step 4: Detect and triage. When an alert fires, the support team decides whether the cause is:
- a data pipeline issue
- an infrastructure issue
- genuine model drift
- a change in business conditions

Step 5: Respond. Possible responses include:
- fixing pipelines
- rolling back
- applying human-in-the-loop overrides
- retraining with fresh data
- in severe cases, sending the work back to earlier CPMAI phases (Data Understanding, Data Preparation, Model Development)

Step 6: Validate and redeploy. A retrained model must pass evaluation against holdout data, business criteria and trustworthy AI checks before it replaces the current version. Common approaches are A/B testing, shadow deployment or canary releases.

Step 7: Document and govern. Update the model registry, audit logs and model cards, and communicate the change.

Step 8: Iterate. Use lessons learned and new business needs to start the next CPMAI iteration. This reflects the cyclical nature of the methodology.

Example Scenario
A retailer deploys a demand-forecasting model. After three months, forecast error rises 20% because consumer behavior shifted. The team responds as follows:
- Monitoring flags the drift alert.
- The support team confirms the data pipelines are healthy.
- The model owner triggers retraining with recent data.
- Validation shows the new model meets the accuracy threshold.
- A canary release confirms improvement, and the new version is promoted.
- The change is documented and planners are informed.

Exam Tips: Answering Questions on Ongoing Maintenance and Support Procedures
1. Remember: AI is never 'done'. Avoid answers that treat deployment as project closure with no monitoring. Prefer answers that build in continuous monitoring and iteration.

2. Monitor before you act. When a question describes falling performance, the best first step is usually to investigate or analyze the cause. That means checking data quality, pipelines and drift. Immediate retraining or replacing the model is often premature.

3. Distinguish drift types.
- Data drift is a change in inputs.
- Concept drift is a change in the input-to-output relationship.
- Pipeline or data quality failures are operational issues, not drift.
Match the response to the cause.

4. Plan early. If asked when to define maintenance procedures, the answer is during planning and pre-deployment, not after problems arise.

5. Tie metrics to business value. CPMAI anchors everything to the business objectives from Phase I. Good answers monitor business KPIs as well as technical metrics.

6. Ownership and accountability. Look for answers that assign a clear model owner, a support structure, SLAs and escalation paths.

7. Governance and trustworthy AI. Correct answers often include:
- ongoing bias and fairness checks
- explainability
- audit trails
- version control
- regulatory compliance
Be wary of options that skip validation or documentation for the sake of speed.

8. Safe redeployment. Retrained models must be validated before production, ideally using shadow, canary or A/B testing, with a rollback plan ready.

9. Human-in-the-loop. For high-risk decisions or when confidence is low, answers that include human oversight or fallback mechanisms are usually preferred.

10. Iteration back to earlier phases. If the problem is fundamental, the CPMAI-aligned answer is to loop back to the appropriate earlier phase rather than patch in production. Examples are new data sources or a changed business question.

11. Watch the keywords. Terms like sustain, monitor, drift, retrain, version, rollback, SLA, feedback loop, governance signal this topic.

12. Eliminate extremes. Discard answers that:
- abandon the model at the first sign of trouble
- ignore stakeholders
- rely entirely on manual ad-hoc checks
- retrain blindly without validation

Quick Summary
Ongoing maintenance and support procedures ensure an operational AI solution keeps delivering business value safely and reliably. Key elements are:
- continuous monitoring of model, data and business metrics
- drift detection
- a defined retraining and validation process
- versioning and governance
- rollback plans
- clear support roles and SLAs
- incident management
- feedback loops
- ongoing trustworthy AI and compliance checks

On the exam, choose answers that are proactive, planned, data-driven, governed, business-aligned and iterative.