Technical Performance Benchmarks and Targets

Overview
In the PMI Cognitive Project Management for AI (CPMAI) methodology, Technical Performance Benchmarks and Targets belong to Phase I, Business Understanding (Identify Business Needs and Solutions). Before any data is collected or any model is trained, the project team must define what "good enough" looks like in measurable technical terms. These benchmarks and targets become the agreed definition of success for the AI solution. They are later checked during Model Evaluation (Phase V) and monitored after Operationalization (Phase VI).

Why It Is Important
1. It prevents endless tuning. AI projects are iterative and probabilistic. Without a target, teams keep tuning models with no clear stopping point. A defined threshold (for example, 92% recall on fraud cases) tells everyone when the model is acceptable.
2. It links technical metrics to business value. A model with 99% accuracy can still be useless. This happens on imbalanced data or when the expensive errors are the ones being missed. Benchmarks force the team to choose metrics that reflect real business outcomes.
3. It supports go/no-go decisions. CPMAI stresses checking feasibility early. If you cannot reach the minimum acceptable performance, or a simpler non-AI approach already performs as well, the project may need to be reframed or stopped. This is the "fail fast, fail cheap" mindset.
4. It sets stakeholder expectations. AI is never 100% accurate. Agreeing on targets up front manages expectations and avoids disappointment and scope disputes later.
5. It enables governance and trustworthy AI. Targets can cover fairness, explainability, latency, robustness and drift tolerance, not just accuracy. This supports responsible AI requirements.
6. It provides the baseline for monitoring. Once the model is deployed, performance degrades through data drift and concept drift. The original benchmarks become the thresholds that trigger retraining.

What It Is
Two related terms make up this topic:
- Benchmark: a reference point used for comparison. Common benchmarks are:
- current human performance
- the existing rules-based system
- industry or published state-of-the-art results
- a simple baseline model, such as predicting the majority class
- competitor performance
- Target: the specific, measurable level the AI solution must achieve to be considered successful. It usually has a minimum acceptable threshold and sometimes a stretch goal.

Technical performance measures usually fall into these categories:
- Model quality metrics:
- Classification: accuracy, precision, recall (sensitivity), specificity, F1 score, AUC-ROC, confusion matrix rates
- Regression: MAE, MSE, RMSE, R-squared
- Ranking or recommendation: precision at k, NDCG
- Natural language and generative tasks: BLEU, ROUGE, perplexity, human ratings
- Operational metrics: latency or response time, throughput, availability, compute cost, scalability, model size (for edge deployment)
- Trustworthiness metrics:
- fairness and bias measures across groups
- explainability requirements
- robustness to noisy or adversarial input
- confidence calibration
- Lifecycle metrics: acceptable drift tolerance, retraining frequency, monitoring thresholds

Technical targets must be distinguished from business KPIs, such as ROI, cost savings, customer satisfaction or reduced processing time. The two are connected but different. A model meeting its F1 target is a technical success. The business KPI shows whether the solution delivered value. CPMAI expects both to be defined in Phase I.

How It Works (Process)
Step 1: Start from the business problem. Confirm the problem truly needs AI, and identify which of the seven patterns of AI applies:
- recognition
- conversational/human interaction
- predictive analytics and decisions
- patterns and anomalies
- personalization/hyper-personalization
- goal-driven systems
- autonomous systems
The pattern points to appropriate metrics. For example, anomaly detection usually emphasizes recall and false-positive rate.
Step 2: Establish the baseline. Measure how the problem is solved today, whether by humans, rules or chance. The AI solution must beat this baseline in a way that matters.
Step 3: Weigh the cost of errors. Work with stakeholders to decide which mistakes are costlier:
- When false negatives are costly, prioritize recall. Examples are missed cancer diagnoses and missed fraud.
- When false positives are costly, prioritize precision. Examples are spam filters blocking important emails and wrongly declined payments.
- When both matter, use F1 or a cost-weighted metric.
Step 4: Select metrics and set thresholds. Define:
- the primary metric or metrics and their minimum acceptable values
- secondary constraints (for example, latency under 200 ms and a fairness gap under 5%)
- the evaluation method, such as a holdout test set, cross-validation or an A/B test
Step 5: Validate feasibility. Check whether the needed data quantity and quality exist to reach the target. Compare against published benchmarks. If the target is unrealistic, reset expectations or rescope.
Step 6: Document and get agreement. Record targets in the project charter or AI project definition, and get sign-off from the business owner, data science team and governance stakeholders.
Step 7: Use iteratively. In Phase V (Model Evaluation), compare results to the targets:
- If targets are met, proceed to operationalization.
- If they are not met, loop back to the data or business understanding phases. This is the iterative, agile nature of CPMAI.
- After deployment, monitor against the targets and trigger retraining when they are breached.

Example
An insurer wants to automate claims triage.
- Benchmark: human adjusters correctly flag 80% of fraudulent claims, and the current rules engine flags 65%.
- Target: recall of at least 85% at a precision of at least 70%, an inference time under 1 second per claim, and no more than a 3% difference in false-positive rate across demographic groups.
- Business KPI: a 20% reduction in fraud losses within 12 months.

Exam Tips: Answering Questions on Technical Performance Benchmarks and Targets
Tip 1: Timing matters. If a question asks when performance targets should be defined, the answer is early, in Phase I Business Understanding, before data preparation and modeling. Be wary of options that set targets after the model is built.
Tip 2: Look past accuracy. Be suspicious of answers that rely on accuracy alone, especially when the scenario mentions rare events (fraud, disease, defects) or imbalanced classes. The better answer usually involves precision, recall, F1 or AUC.
Tip 3: Match the metric to the cost of error. Read the scenario for which mistake is worse.
- Missing a positive is dangerous: choose recall.
- False alarms are expensive: choose precision.
Tip 4: Baselines come first. If asked how to judge whether an AI model is worthwhile, the best answer compares it against a baseline (current process, human performance or a simple model), not an arbitrary number.
Tip 5: Tie targets to business value. PMI favors answers that connect technical metrics to business objectives and stakeholder agreement. Avoid options where the data scientist sets targets alone without business input.
Tip 6: Remember non-functional targets. Latency, scalability, cost, explainability and fairness are valid technical targets. If a scenario involves real-time use, regulated industries or customer-facing decisions, expect these to matter.
Tip 7: Unmet targets mean iterate, not force. When a model misses its target, the CPMAI-aligned response is to iterate by revisiting data or business understanding, or to reassess feasibility. Do not deploy anyway. Lowering the target is acceptable only with stakeholder agreement and a business justification.
Tip 8: Expect imperfection. Answers that demand 100% accuracy or zero errors are almost always wrong. Good answers set realistic, risk-based thresholds and include human oversight where needed.
Tip 9: Targets continue after deployment. Questions about operationalization may test whether you know targets become monitoring thresholds for drift and retraining.
Tip 10: Eliminate extremes. Discard options that:
- skip measurement (for example, "deploy and see")
- use vague goals (for example, "make the model as good as possible")
- ignore stakeholders
Choose answers that are specific, measurable, agreed and tied to the business need.
Tip 11: Watch keywords. Words like "acceptable performance," "success criteria," "definition of done," "go/no-go" and "threshold" signal this topic.

Quick Summary
Define measurable, realistic technical success criteria during Business Understanding. Base them on a benchmark, choose metrics that reflect the real cost of errors, and include operational and trustworthy-AI constraints. Get stakeholder agreement, then use the criteria for evaluation, go/no-go decisions and ongoing monitoring.