Algorithm Selection Criteria and Documentation

Overview
In the PMI Cognitive Project Management for AI (PMI-CPMAI) framework, which is built on the CPMAI methodology, Algorithm Selection Criteria and Documentation sits inside the domain of managing AI model development and evaluation. It covers two things:
1. How a project team decides which algorithm or modeling approach to use.
2. How that decision is recorded so that it is transparent, repeatable, auditable, and aligned with business goals.

The exam does not expect you to be a data scientist. It expects you to be an AI project leader who makes sure the selection is driven by business requirements, data realities, and constraints. You are also responsible for making sure the reasoning is captured properly.

Why It Is Important
1. Business alignment: The most accurate algorithm is not always the right one. A model that cannot be explained, deployed, or maintained may fail the business objective set in Phase I (Business Understanding).
2. Risk reduction: Poor algorithm choices lead to wasted iterations, overfitting, bias, latency problems, and budget overruns.
3. Trustworthy AI: Regulated domains need explainability, fairness, and accountability. Examples include credit, healthcare, hiring, and insurance. The algorithm choice directly affects whether these can be achieved.
4. Reproducibility and auditability: Documentation lets others understand why a model was chosen, reproduce results, and defend the decision to regulators, auditors, or stakeholders.
5. Knowledge transfer and iteration: CPMAI is iterative. Documented decisions let future iterations start from evidence rather than guesswork.
6. Operationalization: The choice affects deployment, monitoring, retraining, and cost in Phase VI (Model Operationalization).

What It Is
Algorithm selection is a structured decision process carried out mainly in CPMAI Phase IV (Model Development). It draws on outputs from:

• Phase I: Business Understanding
• Phase II: Data Understanding
• Phase III: Data Preparation

The team evaluates candidate approaches against defined criteria, then selects one or more, typically after experimentation.

Documentation is the formal record of that decision. It includes the problem framing, candidates considered, criteria used, experiments run, results, trade-offs, the final choice, and its rationale. Common artifacts include:

• Model selection reports
• Experiment logs
• Model cards
• Decision logs
• Assumption logs
• Risk registers

Key Algorithm Selection Criteria
1. Problem type / AI pattern. The CPMAI methodology uses the Seven Patterns of AI:

• Recognition
• Conversational and human interaction
• Predictive analytics and decision support
• Patterns and anomalies
• Hyperpersonalization
• Autonomous systems
• Goal-driven systems

The pattern, plus the task type, narrows the algorithm family. Task types include classification, regression, clustering, sequence generation, and reinforcement learning.
2. Learning paradigm. Supervised, unsupervised, semi-supervised, reinforcement, or self-supervised. This depends on whether labeled data exists.
3. Data characteristics. Volume, variety (structured vs. unstructured), quality, label availability, dimensionality, class imbalance, and data velocity. Deep learning usually needs large data. Smaller tabular datasets often suit tree-based models or linear models.
4. Performance requirements. The metrics defined in Business Understanding, such as accuracy, precision, recall, F1, AUC, RMSE, or MAE. Precision vs. recall depends on the cost of false positives vs. false negatives.
5. Explainability and interpretability. Regulated or high-stakes decisions may favor interpretable models, such as logistic regression or decision trees, or require explainability tools like SHAP or LIME.
6. Fairness, bias, and ethics. Can the approach be tested and mitigated for bias? Does it meet responsible AI principles?
7. Computational resources and cost. Training time, inference latency, hardware such as GPUs, cloud cost, and energy use.
8. Deployment environment. Edge vs. cloud, real-time vs. batch, integration with existing systems, model size.
9. Scalability and maintainability. Ease of retraining, sensitivity to data drift, monitoring needs.
10. Team skills and tooling. The organization's ability to build, support, and govern the model.
11. Time-to-value. Start simple. CPMAI favors small iterations that prove value quickly. A simple baseline model is usually built first.
12. Build vs. buy vs. reuse. Options include pre-trained models, foundation models, transfer learning, or vendor APIs instead of building from scratch.
13. Regulatory and compliance constraints. Examples include GDPR, the EU AI Act, and industry-specific rules.

How It Works (Typical Process)
Step 1: Revisit the business problem and success criteria. Confirm the AI pattern, KPIs, and go/no-go thresholds from Phase I.
Step 2: Review data findings. Use the outputs of Data Understanding and Data Preparation, such as data quality, labels, and features.
Step 3: Shortlist candidate algorithms. Base the list on the problem type and data. Always include a simple baseline, such as a majority-class predictor, linear or logistic regression, or a heuristic rule.
Step 4: Define weighted selection criteria. Agree with stakeholders on how much each criterion matters, for example a decision matrix weighting accuracy, explainability, latency, and cost.
Step 5: Experiment. Train candidates using proper data splits (train, validation, test) or cross-validation. Tune hyperparameters and track every experiment, including data version, parameters, code version, and metrics.
Step 6: Evaluate against criteria. Compare more than performance metrics. Also compare fairness, explainability, robustness, and operational fit.
Step 7: Select and justify. Choose the algorithm that best balances trade-offs. Escalate trade-offs that affect business outcomes to stakeholders.
Step 8: Document. Record the selection rationale, alternatives rejected and why, assumptions, limitations, risks, intended use, and out-of-scope use.
Step 9: Go/no-go and handoff. Feed the results into Phase V (Model Evaluation). If criteria are not met, iterate back to earlier phases.

What Good Documentation Contains

• Business objective and AI pattern
• Data sources, versions, lineage, and preprocessing steps
• Candidate algorithms and the baseline
• Selection criteria and weights
• Experiment results: metrics, confidence, validation method
• Hyperparameters and configuration
• Trade-off analysis and final rationale
• Explainability and fairness assessments
• Assumptions, limitations, and known risks
• Intended use and prohibited uses (as in a model card)
• Approvals and sign-offs, plus version history
• Monitoring and retraining recommendations

Common Pitfalls

• Choosing the newest or most complex algorithm (hype-driven selection)
• Optimizing only for accuracy and ignoring explainability, latency, or cost
• Skipping a baseline
• Evaluating on training data, or leaking test data
• Failing to document rejected options and their reasons
• Selecting the algorithm before understanding the data
• Ignoring stakeholder or regulatory requirements

Exam Tips: Answering Questions on Algorithm Selection Criteria and Documentation
1. Think like a project leader, not a coder. The correct answer usually ties the choice back to business objectives and success criteria, not technical sophistication.
2. Business first, data second, algorithm third. If an option jumps to selecting an algorithm before the problem and data are understood, it is likely wrong.
3. Favor simplicity and baselines. When asked what to do first in model development, building a simple baseline or starting with the simplest viable approach is usually best.
4. Explainability wins in regulated or high-stakes scenarios. In lending, healthcare, hiring, or legal cases, prefer interpretable models or approaches with explainability. This holds even if a black-box model is slightly more accurate.
5. Watch for trade-off keywords. These keywords point to specific criteria:

• "Real-time" or "edge device": latency and model size
• "Limited labeled data": transfer learning, pre-trained models, or unsupervised/semi-supervised approaches
• "Auditors" or "regulators": documentation and explainability
• "Imbalanced classes": appropriate metrics, such as precision, recall, or F1, rather than accuracy

6. Documentation answers are usually comprehensive. When asked what to document, choose the answer that captures rationale, alternatives considered, assumptions, data versions, metrics, and limitations, not just the final model.
7. Reproducibility and traceability. Answers mentioning experiment tracking, versioning data, code, and models, and lineage are strong choices.
8. Involve stakeholders on trade-offs. If a trade-off affects business value or risk, the best answer usually involves consulting or getting approval from stakeholders or the sponsor rather than deciding unilaterally.
9. Iteration is expected. If a model fails to meet criteria, the CPMAI answer is often to iterate back to earlier phases, such as Data Preparation or Business Understanding. Forcing deployment is wrong.
10. Eliminate extreme or absolute options. Distrust choices like "always use deep learning" or "accuracy is the only metric that matters."
11. Responsible AI is part of selection. Fairness, bias testing, privacy, and transparency are selection criteria, not afterthoughts.
12. Consider build vs. buy. If a pre-trained or vendor solution meets requirements faster and cheaper, that is often the preferred answer.
13. Know the phase. Algorithm selection happens in Phase IV (Model Development). Validation against business KPIs happens in Phase V (Model Evaluation). Questions may test whether you know which activity belongs where.

Sample Question Approach
Scenario: A bank wants a loan approval model. The data science team proposes a deep neural network with 2% higher accuracy than a gradient-boosted tree with SHAP explanations. Regulators require decisions to be explainable. What should the project manager do?
Best answer: Select or recommend the explainable approach that meets regulatory requirements. Document the trade-off and rationale, and confirm the choice with stakeholders.
Why: It aligns with business and regulatory constraints, applies explainability as a criterion, and ensures documentation and stakeholder alignment.

Key Takeaway
Algorithm selection in CPMAI is a business-driven, data-informed, criteria-based, and iterative decision. Documentation makes that decision transparent, reproducible, and defensible. On the exam, choose answers that balance performance with explainability, cost, ethics, and operational fit. Make sure the rationale is always recorded.