Evaluating Algorithms for Specific Use Cases

Evaluating Algorithms for Specific Use Cases: A Complete Guide for the PMI-CPMAI Exam

1. Introduction
In the PMI Certified Professional in Managing AI (PMI-CPMAI) framework, which is based on the Cognitive Project Management for AI (CPMAI) methodology, model development is not about building the most sophisticated algorithm. It is about choosing the algorithm that best solves the business problem defined at the start of the project.

Evaluating algorithms for specific use cases sits within the domain Manage AI Model Development and Evaluation. It connects back to the earlier phases: Business Understanding, Data Understanding and Data Preparation. It also feeds forward into Model Evaluation and Model Operationalization.

2. Why It Is Important
Business alignment: An algorithm can have high technical accuracy and still fail if it does not fit the business objective. Examples include a model too slow for real-time decisions, or one too opaque for a regulated environment.
Avoiding wasted investment: AI projects often fail because teams pick complex techniques, such as deep learning, when a simpler method would work. The complex choice increases cost, data requirements and time to value.
Risk and compliance: Use cases in healthcare, finance, hiring and lending often need explainability, fairness and auditability. Algorithm choice directly affects whether the solution can be trusted and legally deployed.
Data realities: The amount, quality, type and labeling of the data limit which algorithms are viable.
Operational feasibility: Inference speed, compute cost, scalability and maintainability determine whether a model survives in production.
Go/no-go decisions: CPMAI stresses iterative checkpoints. Evaluating algorithms against use-case criteria lets the project manager decide whether to proceed, iterate or stop. In some cases the right answer is that AI is not needed at all.

3. What It Is
Evaluating algorithms for specific use cases is the structured process of three steps:
(a) identifying the AI pattern or problem type of the use case;
(b) selecting candidate algorithms suited to that pattern and the available data;
(c) comparing those candidates against technical and business criteria to choose the most appropriate one.

The Seven Patterns of AI are a common CPMAI reference for classifying use cases:
1. Recognition: identifying objects, images, speech or text. Typical algorithms are CNNs and transformers.
2. Conversation and Human Interaction: chatbots, voice assistants and NLP. Typical algorithms are LLMs, transformers and sequence models.
3. Predictive Analytics and Decision Support: forecasting and predicting outcomes. Typical algorithms are regression, decision trees, random forests, gradient boosting and time-series models.
4. Patterns and Anomalies: fraud detection and outlier detection. Typical algorithms are clustering, isolation forests and autoencoders.
5. Hyperpersonalization: recommendations and individual profiling. Typical algorithms are collaborative filtering, matrix factorization and deep recommenders.
6. Autonomous Systems: self-driving vehicles, robotics and agents. Typical algorithms are reinforcement learning combined with perception models.
7. Goal-Driven Systems: optimization, game playing and finding the best path. Typical algorithms are reinforcement learning, search and optimization methods.

Learning approaches:
Supervised learning uses labeled data and is used for classification and regression.
Unsupervised learning uses unlabeled data and is used for clustering, dimensionality reduction and anomaly detection.
Semi-supervised and self-supervised learning use limited labels.
Reinforcement learning learns by trial and error with a reward signal.
Generative AI creates new content such as text, images or code.

4. How It Works: The Evaluation Process

Step 1: Revisit the business problem and success criteria
Go back to the Business Understanding phase. Ask these questions:
- What decision or action will the model support?
- What does success look like? Examples are reduced churn by X%, fewer false fraud alerts, or faster processing.
- What are the constraints? Consider latency, budget, explainability, regulation and ethics.

Step 2: Identify the AI pattern and problem type
Classify the use case. Decide whether it is classification, regression, clustering, anomaly detection, recommendation, generation or optimization. Then map it to one or more of the seven patterns. A single project may combine patterns.

Step 3: Assess data characteristics
- Labeled or unlabeled? This determines supervised versus unsupervised approaches.
- Volume: Deep learning typically needs large datasets. Simpler models work with less data.
- Data type: Structured or tabular data often suits tree-based models. Images suit CNNs. Text and sequences suit transformers and RNNs.
- Quality and balance: Imbalanced classes, such as fraud cases, affect algorithm and metric choice.
- Dimensionality and feature types.

Step 4: Shortlist candidate algorithms
Start with a baseline, such as a simple rule-based approach, logistic regression or a mean predictor. Then add more advanced candidates. CPMAI supports starting simple and iterating, the 'think big, start small, iterate often' mindset.

Step 5: Define evaluation criteria

Technical performance metrics:
- Classification: accuracy, precision, recall (sensitivity), F1-score, ROC-AUC, PR-AUC and the confusion matrix.
- Regression: MAE, MSE, RMSE and R-squared.
- Clustering: silhouette score and the Davies-Bouldin index.
- Recommendation: precision@k, recall@k, NDCG and hit rate.
- Generative and NLP: BLEU, ROUGE, perplexity and human evaluation.

Non-functional and business criteria:
- Explainability and interpretability: Decision trees and linear models are more transparent than deep neural networks.
- Fairness and bias: Check performance across demographic groups.
- Latency and throughput: Real-time scoring versus batch processing.
- Computational cost: Training and inference resources.
- Scalability and maintainability.
- Robustness: Sensitivity to noise and drift.
- Regulatory compliance: For example GDPR, the EU AI Act and sector rules.
- Cost of errors: The business impact of false positives versus false negatives.

Step 6: Choose metrics based on the cost of errors
This is a heavily tested concept:
- Recall is prioritized when missing a positive is costly. Examples are cancer screening, fraud detection and safety defects.
- Precision is prioritized when false alarms are costly. Examples are spam filtering of important emails and wrongly flagging legitimate transactions that irritate customers.
- F1-score balances precision and recall, especially with imbalanced classes.
- Accuracy is misleading on imbalanced data. A model that predicts 'no fraud' 99% of the time can look 99% accurate while catching nothing.

Step 7: Train, validate and compare
- Use proper data splitting into train, validation and test sets, or use cross-validation.
- Prevent data leakage.
- Watch for overfitting, where the model does well on training data and poorly on test data. Also watch for underfitting, where it does poorly on both.
- Tune hyperparameters.
- Compare candidates against the baseline using the agreed metrics.

Step 8: Evaluate trade-offs and make a decision
Weigh accuracy against interpretability, performance against cost, and complexity against maintainability. Involve stakeholders such as business owners, compliance, data scientists and operations. Document the rationale.

Step 9: Go/No-Go and iterate
If no candidate meets business criteria, return to earlier phases. That may mean collecting more data, reframing the problem or adjusting scope. CPMAI is iterative, not waterfall.

5. Common Algorithm Trade-offs to Know
Linear and logistic regression: simple, fast and highly interpretable. May underfit complex patterns.
Decision trees: interpretable. Prone to overfitting.
Random forest and gradient boosting (e.g., XGBoost): strong on tabular data. Moderately interpretable through feature importance and SHAP.
Support vector machines: effective in high dimensions. Less scalable to very large data.
K-means clustering: simple segmentation. Requires choosing k in advance.
Neural networks and deep learning: excellent for images, speech and text. Need large data and compute, and are less explainable ('black box').
Transformers and LLMs: powerful for language and generative tasks. Costly, can hallucinate, and raise data privacy concerns.
Reinforcement learning: suited to sequential decisions. Needs simulation environments and careful reward design.

6. Role of the AI Project Manager
The project manager does not have to code the models. They must:
- ensure algorithm choices trace back to business objectives;
- confirm that evaluation metrics reflect business value and risk;
- facilitate stakeholder trade-off decisions;
- ensure responsible and trustworthy AI considerations are included;
- manage iteration, scope and go/no-go checkpoints;
- make sure decisions and model documentation (e.g., model cards) are recorded.

7. Exam Tips: Answering Questions on Evaluating Algorithms for Specific Use Cases

Tip 1: Always anchor to the business objective.
When a question asks which algorithm or approach is 'best', the correct answer is usually the one that best meets the stated business need and constraints. It is rarely the most advanced or most accurate option. Look for keywords like regulated, real-time, limited data, must explain decisions or low budget.

Tip 2: Explainability requirements favor interpretable models.
If the scenario mentions regulators, auditors, loan denial, medical decisions or customer right-to-explanation, prefer interpretable models. Examples are logistic regression and decision trees, or complex models paired with explainability tools such as SHAP and LIME. Be wary of answers choosing an unexplainable deep neural network without justification.

Tip 3: Match metrics to the cost of errors.
- Missing a positive is dangerous (disease, fraud, safety): choose recall.
- False alarms are expensive or harmful: choose precision.
- Classes are imbalanced and both errors matter: choose F1-score or PR-AUC.
- Never default to accuracy on imbalanced data.

Tip 4: Identify the AI pattern first.
Translate the scenario into one of the seven patterns. 'Detect unusual transactions' is Patterns and Anomalies. 'Recommend products' is Hyperpersonalization. 'Forecast demand' is Predictive Analytics. This narrows the algorithm choices fast.

Tip 5: Check data availability and labeling.
- No labels: consider unsupervised methods such as clustering or anomaly detection.
- Small dataset: avoid data-hungry deep learning and prefer simpler models or transfer learning.
- If the data cannot support the use case, the right answer may be to return to Data Understanding or Data Preparation.

Tip 6: Prefer starting simple with a baseline.
CPMAI promotes iterative, incremental delivery. Answers that establish a baseline and iterate are usually preferred over building a complex model from day one.

Tip 7: Recognize overfitting and underfitting scenarios.
- High training performance with low test performance means overfitting. Remedies include regularization, more data, simpler models and cross-validation.
- Low performance on both means underfitting. Remedies include more features, more complex models and better data.

Tip 8: Consider operational constraints.
If the scenario requires millisecond responses or edge deployment, favor lightweight models. If the cost of compute is a concern, avoid recommending large models without justification.

Tip 9: Don't forget responsible AI.
Answers that include bias testing, fairness across groups, transparency and human oversight are often correct, especially for high-impact use cases.

Tip 10: Know when the answer is 'go back'.
If no model meets success criteria, the correct action is often to revisit earlier CPMAI phases rather than force deployment. Deploying an underperforming model to meet a deadline is almost always wrong.

Tip 11: Involve stakeholders in trade-offs.
When asked what the project manager should do about a trade-off, such as accuracy versus explainability, the best answer typically involves consulting business owners and stakeholders and documenting the decision. The project manager should not decide unilaterally.

Tip 12: Eliminate distractors.
Remove answers that:
- pick an algorithm because it is trendy;
- ignore stated constraints;
- skip validation or testing;
- use the test set for training or tuning (data leakage);
- recommend deployment without meeting acceptance criteria.

8. Sample Exam-Style Questions

Q1: A bank wants to automate loan approval decisions. Regulators require that applicants receive a reason for denial. Two models were tested: a deep neural network with 94% accuracy and a gradient-boosted model with 92% accuracy and SHAP explanations. What should the project manager recommend?
Answer: The gradient-boosted model with explanations. It satisfies the regulatory explainability requirement with only a minor accuracy trade-off.

Q2: A hospital model screens for a rare but serious disease. Which metric should be prioritized?
Answer: Recall (sensitivity). Missing a true case (false negative) is far more costly than a false positive that leads to further testing.

Q3: A retailer has years of transaction data but no labels and wants to discover customer segments. Which approach fits?
Answer: Unsupervised clustering, such as k-means or hierarchical clustering.

Q4: A model performs at 98% on training data and 70% on test data. What is the issue and what should be done?
Answer: Overfitting. Apply regularization, simplify the model, gather more data or use cross-validation.

Q5: After several iterations, no candidate model meets the business success threshold. What is the best next step?
Answer: Hold a go/no-go review. Revisit Business Understanding or Data Understanding and Data Preparation, which may mean reframing the problem or acquiring better data, rather than deploying a substandard model.

9. Key Takeaways
- Algorithm evaluation is driven by the business use case, not technical novelty.
- Classify the problem using the seven patterns of AI and the learning type.
- Let data characteristics constrain algorithm choice.
- Select metrics aligned with the cost of errors, and beware of accuracy on imbalanced data.
- Balance performance, explainability, fairness, cost and latency.
- Start with a baseline and iterate in line with CPMAI's agile, data-centric approach.
- Use go/no-go checkpoints and be willing to return to earlier phases.
- In the exam, choose the answer that is business-aligned, responsible, data-feasible and iterative.