Deployment Checklists and Validation Criteria

Overview
In the PMI Cognitive Project Management for AI (CPMAI) methodology, the final phase, Operationalize the AI Solution (Phase VI), turns a validated model into a working, trusted business capability. Two tools govern this transition: Deployment Checklists and Validation Criteria. Together they make sure a model is not just technically accurate but also ready for production, compliant, monitored and actually delivering the business value set in Phase I (Business Understanding).

Why It Is Important
Many AI projects fail at the last mile. Models that score well in the lab often fail in production because of:
- Data drift and concept drift
- Infrastructure mismatches
- Missing monitoring
- Unclear ownership
- Regulatory gaps
- Lack of user adoption

Deployment checklists and validation criteria reduce this risk in several ways:
- Risk reduction: Issues are caught before they affect customers, revenue or compliance.
- Business alignment: The solution is confirmed to meet the success criteria defined at the start (for example, reduce fraud losses by 15%), not just a technical metric.
- Trustworthy AI: Fairness, transparency, explainability, privacy and security requirements are verified before go-live.
- Repeatability: A standard process supports MLOps practices and iterative CPMAI cycles.
- Accountability: Clear go/no-go decisions are made with documented sign-off from stakeholders.
- Sustainability: Monitoring, retraining triggers and support are in place, because AI systems degrade over time.

What It Is
Deployment Checklist: A structured, itemized list of conditions that must be met before, during and after moving a model into production. It covers technical, operational, organizational, legal and ethical readiness.

Validation Criteria: The measurable thresholds and qualitative standards used to judge whether the model and the whole solution are fit for purpose. They fall into these groups:
- Technical performance: accuracy, precision, recall, F1, AUC, RMSE, latency, throughput.
- Business KPIs: ROI, cost savings, conversion lift, reduced cycle time.
- Operational criteria: uptime, scalability, integration success, response time SLAs.
- Trustworthy AI criteria: bias and fairness metrics, explainability, privacy compliance, security testing.
- User acceptance: end-user satisfaction and adoption rates.

Validation criteria should be set early (Phases I and IV), not invented at deployment time. In CPMAI, Phase IV (Model Evaluation) checks the model against these criteria. Phase VI confirms they still hold in the real operating environment and continue to hold over time.

How It Works
1. Define acceptance criteria up front. During Business Understanding, the team sets the definition of success, the go/no-go thresholds and the acceptable risk levels. These later become the validation criteria.

2. Build the deployment checklist. Typical items include:
- Model version, artifacts and documentation (model cards, data lineage) are complete and under version control.
- The production data pipeline is tested, and its data matches the training data schema and quality.
- Infrastructure is provisioned (cloud, edge, on-premises) with scalability and failover tested.
- APIs and integration with business systems are tested end to end.
- Security review, access controls and privacy (GDPR, HIPAA and similar) are verified.
- Bias, fairness and explainability assessments are completed and approved.
- Monitoring is configured for model performance, data drift, concept drift and system health.
- Alerting thresholds and retraining triggers are defined.
- A rollback plan exists, and the previous version or fallback process is available.
- Human-in-the-loop and escalation procedures are documented.
- Users are trained, change management is done and support handoff is complete.
- Governance sign-off has been given by business owners, compliance and IT operations.

3. Choose a deployment strategy. Common options are:
- Shadow deployment: the model runs silently alongside the current process.
- Canary release: the model serves a small percentage of traffic first.
- A/B testing
- Blue-green deployment
- Phased rollout

Each strategy lets the team validate against criteria in real conditions before full release.

4. Validate in production. The team compares live performance against baseline and threshold criteria. It confirms that business KPIs are trending as expected and checks that real-world data distributions match training data.

5. Make the go/no-go decision. Stakeholders review checklist completion and validation results. Unmet criteria lead to remediation, rollback or a return to earlier CPMAI phases (for example, Data Preparation or Model Development). CPMAI is iterative.

6. Monitor and revalidate continuously. Validation does not end at launch. Drift monitoring, periodic audits and retraining keep the model within criteria. Models can be retired when they no longer deliver value.

Example
A bank deploys a loan-default prediction model. Its validation criteria are:
- AUC of at least 0.85
- Disparate impact ratio between 0.8 and 1.25 across protected groups
- Latency under 200 ms
- A 10% reduction in default losses within six months

The checklist confirms that explainability reports exist for regulators, that a human reviews borderline cases, and that drift alerts fire if applicant demographics shift. The team uses a shadow deployment for 30 days before the canary release.

Exam Tips: Answering Questions on Deployment Checklists and Validation Criteria
- Business value comes first: When an option ties validation to the original business objectives and KPIs, it usually beats options focused only on technical accuracy. CPMAI stresses that AI projects are business projects.
- Criteria are defined early: Be wary of answers suggesting that success criteria be set at deployment. The best answer usually refers back to criteria set in Business Understanding.
- Monitoring is mandatory: If an answer treats deployment as the end of the project, it is likely wrong. Look for continuous monitoring, drift detection and retraining plans.
- Trustworthy AI is a gate: Bias, fairness, explainability, privacy and security checks are required items, not optional extras. Questions about regulated industries often hinge on these.
- Prefer controlled rollouts: For high-risk scenarios, choose shadow, canary or phased deployment over a 'big bang' release.
- A rollback plan is essential: Any good checklist includes a fallback or rollback mechanism.
- Iteration is expected: If validation fails, the correct action is often to return to a prior CPMAI phase. Forcing deployment or simply lowering the thresholds is usually wrong.
- Stakeholder sign-off: Go/no-go decisions involve business owners, compliance and operations, not just data scientists.
- Watch for performance gaps: A model that met lab metrics but underperforms in production signals data drift, training-serving skew or pipeline issues. Choose answers that investigate data first.
- Read qualifiers carefully: Words like FIRST, BEST and MOST important matter. Before deployment, the first step is usually confirming the solution meets agreed validation and acceptance criteria.
- Eliminate extremes: Discard answers that skip testing, ignore users or rely on a single metric.

Key Takeaway
Deployment checklists confirm that a solution is ready. Validation criteria confirm that it works and delivers value, both at launch and over time. In CPMAI, operationalization is a governed, iterative, business-driven process that keeps AI solutions reliable, compliant and valuable in the real world.