Procurement Due Diligence for AI Solutions

Why Procurement Due Diligence for AI Solutions Matters

Most organizations do not build the AI systems they use. They buy them, license them, embed them through APIs, or inherit them inside larger software platforms. This means that a large share of an organization's AI risk originates outside its own walls, in vendor models, third-party training data, foundation model providers, and cloud services the organization does not directly control. Procurement due diligence is the control point where the organization decides whether to accept that risk, and on what terms.

Within the ISACA Advanced in AI Risk (AAIR) framework, procurement due diligence sits in the AI Life Cycle Risk Management domain because the decision to acquire an AI solution is a life cycle event in its own right. A poorly vetted acquisition injects risk into every subsequent phase: deployment, monitoring, incident response, and retirement. Conversely, strong due diligence allows the organization to establish contractual rights, transparency obligations, and exit provisions before it becomes dependent on the vendor.

Why it matters in practice:
• Opacity of third-party models: The buyer often cannot inspect training data, model weights, or evaluation methods, so it must rely on documentation, attestations, and contractual commitments.
• Regulatory accountability does not transfer: Under frameworks such as the EU AI Act, the deployer retains obligations even when the model was built by someone else. A buyer cannot outsource liability simply by outsourcing development.
• Concentration and lock-in risk: Dependence on a single foundation model provider or vendor can create operational resilience problems that are expensive to unwind.
• Hidden sub-processors and supply chains: An AI vendor may itself rely on upstream model providers, data brokers, and cloud infrastructure, creating fourth-party and nth-party risk.
• Data protection and intellectual property exposure: Prompts, uploaded documents, and outputs may be used to retrain vendor models unless contractually prohibited.

What Procurement Due Diligence for AI Solutions Is

Procurement due diligence for AI solutions is the structured, risk-based process of evaluating a prospective AI vendor, product, or component before acquisition and contracting, and of defining the ongoing obligations that will govern the relationship afterward. It extends traditional third-party risk management (TPRM) with AI-specific dimensions that conventional software vendor assessments do not cover.

Conventional vendor due diligence asks: Is the vendor financially stable? Is their security posture adequate? Do they comply with applicable laws? AI due diligence adds a further set of questions:
• What data was the model trained on, and does the vendor have rights to use it?
• How was the model tested for accuracy, bias, robustness, and safety?
• What are the documented limitations and intended uses?
• How will the model change over time, and how will the buyer be notified?
• What happens to the buyer's data, prompts, and outputs?
• Who is accountable when the model produces harmful or incorrect results?
• Can the solution be explained, audited, and monitored?
• What are the exit options if the vendor fails, is acquired, or discontinues the model?

The scope of AI procurement due diligence includes several categories of acquisition:
1. Complete AI applications (for example, a vendor hiring-screening tool or fraud detection platform).
2. Foundation models and APIs (large language models, vision models, embedding services).
3. AI components embedded in broader software (an AI feature inside a CRM or ERP system, sometimes activated by default without a separate procurement decision).
4. Pre-trained models and datasets obtained from open-source repositories or marketplaces, which may bypass formal procurement entirely and therefore require specific governance.
5. AI development services (consultancies building custom models) where the buyer may own the result but must scrutinize the methods used.

How Procurement Due Diligence for AI Solutions Works

A mature process typically moves through the following stages. Exam questions frequently test whether you can place an activity in the correct stage and identify what should happen first.

Stage 1: Need definition and use-case risk classification
Before approaching the market, the organization should define the business problem, the intended use, and the decisions the AI will influence. The use case is then classified by risk level, often aligned to regulatory tiers (such as prohibited, high-risk, limited-risk, and minimal-risk categories) and to the organization's own AI risk appetite. The classification determines how deep the due diligence must go. A chatbot answering FAQs about office hours warrants a lighter review than a model influencing credit, employment, healthcare, or safety decisions. Due diligence intensity should be proportionate to risk, a principle that appears repeatedly in AAIR material.

Stage 2: Inventory and scope identification
The organization determines whether the acquisition involves a standalone AI system, an embedded capability, or a component. It identifies sub-processors and upstream model providers. A key control here is ensuring the acquisition will be recorded in the organization's AI inventory or register so that it is visible to governance, monitoring, and audit functions. Shadow AI, where business units adopt AI tools without procurement involvement, is a well-known failure mode at this stage.

Stage 3: Vendor and solution assessment
This is the core of due diligence. Typical areas of inquiry include:
• Governance and accountability: Does the vendor have an AI governance framework, responsible AI policies, named accountable roles, and an ethics review process?
• Model documentation: Request model cards, system cards, datasheets for datasets, or equivalent transparency artifacts describing intended use, training data provenance, known limitations, and evaluation results.
• Data provenance and rights: Confirm that training data was lawfully obtained, that personal data was processed with a valid legal basis, and that intellectual property rights are not infringed. Ask whether copyrighted material was used and whether the vendor offers indemnification.
• Performance and testing evidence: Review accuracy metrics, fairness and bias testing across relevant demographic groups, robustness testing, adversarial and red-team results, and performance on data representative of the buyer's own population rather than only on vendor benchmarks.
• Security: Evaluate protections against AI-specific threats such as prompt injection, data poisoning, model inversion, membership inference, and model theft, in addition to conventional application and infrastructure security. Consider independent attestations (SOC 2, ISO/IEC 27001) and AI-specific standards such as ISO/IEC 42001 for AI management systems and alignment with the NIST AI Risk Management Framework.
• Privacy and data handling: Determine where data is processed and stored, whether the buyer's inputs and outputs are retained, whether they are used to train or fine-tune vendor models, and how deletion requests are honored. Data residency and cross-border transfer requirements should be verified.
• Explainability and transparency: Assess whether the solution can provide explanations suitable for the decision context, including for affected individuals and regulators where required.
• Human oversight features: Confirm that the solution supports human-in-the-loop or human-on-the-loop controls, override capabilities, and confidence indicators appropriate to the risk level.
• Change management and model drift: Understand how often the vendor updates the model, whether updates are versioned, whether the buyer is notified in advance, and whether the buyer can pin to a specific version or test updates before adoption. Silent model updates are a significant risk because they can invalidate prior validation.
• Operational resilience: Evaluate availability commitments, redundancy, business continuity planning, and the vendor's financial viability and ownership structure.
• Regulatory and legal compliance: Confirm conformity assessments, registrations, and compliance obligations relevant to the jurisdiction and sector, as well as the vendor's capacity to support the buyer's own compliance obligations such as impact assessments and record-keeping.

Stage 4: Independent verification
Where risk justifies it, the buyer should move beyond vendor self-attestation. Options include proof-of-concept testing on the buyer's own data, independent third-party audits or algorithmic assessments, penetration testing and red-teaming, and reference checks with existing customers. The principle is trust but verify: documentation establishes claims, testing confirms them.

Stage 5: Contracting and risk allocation
Due diligence findings should flow directly into contractual terms. Important AI-specific clauses include:
• Permitted and prohibited uses, aligned to the vendor's stated intended use.
• Data ownership, confidentiality, and an explicit prohibition or opt-out regarding use of buyer data for vendor model training.
• Transparency and notification obligations for material model changes, incidents, and retraining.
• Right to audit, right to obtain documentation, and cooperation with regulators.
• Performance warranties and service levels covering accuracy, availability, and bias thresholds where measurable.
• Intellectual property warranties and indemnification for infringement claims arising from training data or outputs.
• Liability allocation for harmful outputs and for failures of the AI component.
• Security and incident reporting requirements with defined timelines.
• Sub-processor disclosure and approval rights.
• Exit, transition, and data return or deletion provisions, including portability of fine-tuned models or embeddings where applicable.

Stage 6: Onboarding and ongoing monitoring
Due diligence does not end at signature. The solution should be entered into the AI inventory, assigned an owner, and subjected to continuous monitoring for performance, drift, bias, incidents, and vendor compliance. Periodic reassessment should be triggered by time (annual review), by events (major model updates, incidents, regulatory changes, vendor acquisition), or by changes in how the buyer uses the system. Reassessment is particularly important when a use case expands beyond the originally approved scope.

Key Roles and Responsibilities
• Business owner: defines the need, owns the use case, and is accountable for outcomes.
• Procurement: manages the sourcing process and ensures due diligence gates are enforced before contracting.
• AI risk or governance function: performs or coordinates the AI-specific risk assessment and sets proportionality criteria.
• Legal and privacy: address contractual terms, regulatory classification, data protection, and IP.
• Information security: evaluates technical and AI-specific security controls.
• Internal audit: provides independent assurance that the process operates as designed.
• Senior management and the board: set risk appetite and approve high-risk acquisitions.
Separation of duties matters: the business unit requesting the tool should not be the sole party evaluating its risk.

Common Pitfalls
• Treating AI procurement as ordinary software procurement and skipping model-specific questions.
• Relying solely on vendor marketing claims or generic benchmark scores.
• Failing to test on representative data from the buyer's own environment.
• Overlooking embedded AI features activated inside existing platforms.
• Accepting contracts that permit the vendor to train on buyer data by default.
• Not negotiating notification rights for model changes.
• Ignoring the vendor's upstream dependencies on foundation model providers.
• Having no exit strategy, leading to lock-in.
• Allowing due diligence findings to be documented but never acted upon in the contract or in deployment controls.

How to Answer Exam Questions on Procurement Due Diligence for AI Solutions

AAIR questions in this area are typically scenario-based. You will be given a situation (an organization considering a vendor AI tool, a model update, a contract negotiation, a discovered gap) and asked what the best, first, most important, or primary action or concern is. Several answer options will be plausible; your task is to select the one that best reflects risk management principles.

Exam Tips: Answering Questions on Procurement Due Diligence for AI Solutions

1. Proportionality wins. If a question asks how much due diligence to perform, the correct answer almost always ties the depth of assessment to the risk classification of the use case, not to the cost of the solution or the size of the vendor.

2. Accountability stays with the deployer. Any option implying that purchasing from a vendor transfers regulatory or ethical responsibility away from the organization is a distractor. The buyer remains accountable for how the AI is used.

3. Sequence matters: classify, then assess, then contract, then monitor. When asked what should be done first, look for the option that defines the use case and its risk level or identifies requirements before evaluating vendors. Jumping straight to contract negotiation or technical testing without establishing intended use and risk tier is usually wrong.

4. Prefer verification over attestation. When choosing between accepting vendor documentation and independently validating claims (testing on the buyer's data, third-party audit, red-teaming), the higher-risk the scenario, the more the exam favors independent verification.

5. Documentation artifacts are a signal. Recognize terms such as model cards, system cards, datasheets for datasets, and transparency notes. An answer that requests these artifacts to understand intended use, training data, and limitations is typically a strong choice.

6. Watch for data-use clauses. If a scenario involves confidential or personal data being sent to a vendor model, the best answer usually involves contractually restricting the vendor from using inputs and outputs for training and confirming data residency, retention, and deletion terms.

7. Model change notification is a recurring theme. Scenarios about a vendor silently updating a model point toward answers involving contractual change-notification rights, version pinning, re-validation before adoption, and ongoing monitoring.

8. Think about the whole supply chain. If the vendor relies on an upstream foundation model provider, the best answer accounts for fourth-party risk and sub-processor transparency rather than stopping at the direct vendor.

9. Contracts should operationalize findings. An answer that translates identified risks into specific contractual protections (audit rights, incident notification, IP indemnification, exit provisions) is usually stronger than one that merely documents the risk.

10. Due diligence is continuous. Options suggesting a one-time assessment at purchase are weaker than options that include ongoing monitoring and periodic or event-triggered reassessment.

11. Do not forget the inventory. Questions about shadow AI, embedded features, or unknown AI usage often resolve to establishing or updating the AI inventory and routing acquisitions through governance gates.

12. Eliminate extremes. Options that say the organization should never use third-party AI, or should always accept vendor assurances, or should rely exclusively on legal remedies rather than controls, are almost always distractors. Balanced, risk-based, governance-aligned answers are favored.

13. Identify the role being asked about. If the question is framed from the perspective of an AI risk professional, choose the answer that reflects risk assessment and advisory responsibility rather than business decision-making or technical implementation that belongs to other roles.

14. Read for the keyword. Terms like primary, most effective, best, and first each change the correct answer. Most effective typically favors preventive or verifiable controls; first favors scoping and classification; primary concern favors the risk with the greatest impact on the organization's accountability or on affected individuals.

Sample Question Walkthrough

An organization plans to license a third-party AI tool to screen job applicants. The vendor provides a brochure stating the model is highly accurate and unbiased. What should the AI risk professional recommend FIRST?

A. Negotiate liability caps in the contract.
B. Classify the use case as high-risk and define the required assessment scope.
C. Deploy the tool in a pilot to observe outcomes.
D. Accept the vendor's statement and proceed with procurement.

Option D ignores verification. Option A puts contracting ahead of assessment. Option C tests before scope and requirements are defined and could expose real applicants to an unvalidated system. Option B establishes the risk tier (employment decisions are commonly designated high-risk), which drives the depth of due diligence, the evidence required (bias testing by demographic group, documentation, independent validation), and the contractual terms. B is correct.

Summary

Procurement due diligence for AI solutions is the mechanism by which an organization extends its AI risk management across its supply chain. It is risk-proportionate, begins with use-case classification, relies on both documentation and independent verification, converts findings into enforceable contractual protections, and continues through the operational life of the solution. On the AAIR exam, favor answers that preserve organizational accountability, sequence activities logically, verify rather than assume, address data and model-change risks explicitly, and treat due diligence as an ongoing life cycle control rather than a one-time checkpoint.