Accountability for Third-Party and Embedded AI

1. Why This Topic Matters

Most organizations today do not build the majority of their AI from scratch. They buy it, license it, subscribe to it, or inherit it inside other products. A CRM platform quietly adds a generative assistant. A recruiting tool uses a vendor-built scoring model. A cloud provider offers a foundation model through an API. A medical device ships with embedded machine learning. In every one of these cases, the organization using the AI did not design the model, did not choose the training data, and may not even be able to see how the model makes decisions.

This creates a fundamental governance problem: you can outsource the technology, but you cannot outsource the accountability. Regulators, courts, customers, and boards hold the deploying organization responsible for the outcomes the AI produces. If a vendor model discriminates against job applicants, the employer faces the lawsuit. If an embedded AI feature leaks personal data, the data controller faces the regulator. The EU AI Act, for example, assigns distinct but overlapping obligations to providers, deployers, importers, and distributors, and a deployer who substantially modifies a system can even become a provider.

For the ISACA Advanced in AI Risk (AAIR) exam, this topic sits inside the AI Risk Governance and Framework Integration domain because it tests whether you understand how governance structures, risk ownership, and framework controls extend beyond the organizational boundary to cover AI the organization did not create. It is one of the most practically important and most frequently misunderstood areas of AI risk management.

2. What It Is: Core Concepts and Definitions

Third-party AI refers to AI systems, models, components, or services acquired from an external party. This includes:
• Commercial AI products and SaaS platforms with AI functionality
• Foundation models and large language models accessed through APIs
• Pre-trained models or model weights downloaded from repositories
• AI development services and outsourced data labeling
• Open-source AI libraries, frameworks, and datasets

Embedded AI refers to AI capability that is integrated within a larger product, platform, or service, often invisibly. The AI is not the product being purchased; it is a feature inside it. Examples include fraud detection inside a payment gateway, predictive text inside an email client, autonomous functions inside industrial equipment, or recommendation engines inside an e-commerce platform. Embedded AI is especially dangerous from a governance perspective because it frequently enters the organization without anyone recognizing it as AI, meaning it bypasses AI-specific risk assessment entirely.

Accountability in this context means that a clearly identified role within the deploying organization is answerable for the risks, outcomes, and compliance of the AI system, regardless of who built it. Accountability is distinct from responsibility (who performs the work) and from liability (who bears legal or financial consequences). A vendor may be contractually liable for defects, but the deploying organization remains accountable to its regulators, customers, and board for how the system is used.

Shared responsibility is the principle that AI risk is distributed across the supply chain. The provider is typically responsible for model design, training data governance, safety testing, and documentation. The deployer is responsible for appropriate use, context-specific risk assessment, human oversight, input data quality, monitoring in production, and informing affected individuals. The boundary between these responsibilities must be explicit, documented, and understood by both parties.

3. How It Works: The Accountability Lifecycle for Third-Party and Embedded AI

Step 1: Discovery and Inventory
You cannot govern what you do not know exists. The first control is a complete AI inventory (sometimes called an AI register or AI bill of materials) that captures every AI system in use, including embedded AI inside procured products. Effective discovery techniques include reviewing software contracts and terms of service for AI clauses, scanning vendor release notes for new AI features, questioning business units during risk assessments, analyzing API traffic to known AI endpoints, and adding AI-identification questions to procurement intake forms. Shadow AI, meaning employees using unapproved AI tools, must also be captured.

Step 2: Classification and Risk Tiering
Each third-party or embedded AI system is classified by its risk level based on the use case, the data it processes, the decisions it influences, the affected population, and regulatory applicability. A chatbot answering FAQs about store hours is low risk; a vendor model deciding credit limits is high risk. Risk tiering determines the depth of due diligence, contractual rigor, and ongoing monitoring required. Frameworks such as the NIST AI Risk Management Framework (Map, Measure, Manage, Govern), ISO/IEC 42001, and the EU AI Act risk categories all support this tiering approach.

Step 3: Due Diligence and Vendor Assessment
Before acquisition, the organization evaluates the provider against AI-specific criteria beyond traditional vendor security reviews. Key areas include:
• Transparency documentation: model cards, system cards, datasheets for datasets, intended use statements, and known limitations
• Training data provenance: sources, consent, copyright and licensing, representativeness, and personal data handling
• Performance and fairness evidence: accuracy metrics, bias testing results, robustness evaluations, and red-teaming outcomes
• Security posture: protection against prompt injection, model extraction, data poisoning, and adversarial inputs
• Governance maturity: whether the provider has an AI management system, independent audits, or certifications such as ISO/IEC 42001
• Change management: how and when the provider updates models, and whether customers are notified before behavior changes
• Data usage: whether customer inputs are used to retrain the vendor model, and whether data residency and retention commitments exist

Step 4: Contractual Allocation of Responsibility
The contract is the primary mechanism for converting shared responsibility into enforceable obligations. Essential AI-specific contract provisions include: right to audit or receive third-party assurance reports; notification of material model changes before deployment; performance and accuracy service levels with remedies; data ownership and prohibition on training with customer data without consent; incident notification timelines for AI failures or security events; compliance warranties for applicable AI regulations; indemnification for intellectual property infringement in model outputs; exit and transition rights including access to logs and the ability to switch providers; and clear statements of intended use and prohibited use. Where the vendor refuses these terms, the organization must document the residual risk and obtain formal acceptance from the accountable executive.

Step 5: Internal Accountability Assignment
Every third-party and embedded AI system must have a named business owner accountable for its outcomes, a technical owner responsible for integration and monitoring, and a risk or compliance owner responsible for oversight. This is frequently formalized using a RACI matrix. The business owner, not the vendor and not the IT department, is accountable for ensuring the system is used within its intended scope and that human oversight is operating. Governance committees, such as an AI ethics board or AI risk committee, provide escalation and approval authority for high-risk systems.

Step 6: Integration Controls and Human Oversight
The deployer controls the context in which the AI operates. Controls include input validation and data quality checks, output filtering and guardrails, human-in-the-loop or human-on-the-loop review for consequential decisions, usage restrictions preventing the system from being applied outside its validated scope, access controls, and logging of inputs and outputs to support auditability and explainability requests. For embedded AI, the organization may need to configure or disable AI features it has not assessed.

Step 7: Continuous Monitoring
Third-party models change. Vendors retrain, fine-tune, and replace models, sometimes without notice, and model behavior can drift as real-world data shifts. Ongoing monitoring tracks performance metrics, fairness indicators, error rates, user complaints, and vendor change notifications. Concentration risk is also monitored: dependence on a single foundation model provider across many business processes creates systemic exposure. Periodic reassessment, at least annually and upon any material change, keeps the risk rating current.

Step 8: Incident Response and Exit
Incident response plans must address AI-specific scenarios such as harmful outputs, bias discovered in production, vendor model outages, data leakage through prompts, and regulatory inquiries about a vendor system. Plans define how the organization will obtain information from the vendor, who communicates with regulators and affected individuals, and when the system will be suspended. Exit planning ensures the organization can continue operating if the vendor fails, is acquired, or becomes non-compliant.

4. Common Failure Modes

• Treating AI procurement as ordinary software procurement and skipping AI-specific due diligence
• Assuming the vendor is accountable because the vendor built the model
• Allowing embedded AI features to activate automatically after a product update without reassessment
• Accepting vendor claims of fairness or accuracy without evidence or independent validation
• Failing to restrict the AI to its validated intended use, so a tool approved for one purpose spreads to higher-risk purposes
• Having no named internal owner, so when something goes wrong everyone points to the vendor
• Ignoring the fourth-party risk created when a vendor itself relies on an upstream foundation model provider
• Permitting customer or employee data to flow into vendor training sets without consent or legal basis

5. How Frameworks Address This Topic

The NIST AI RMF Govern function explicitly calls for policies addressing third-party AI, including risks from third-party software and data. The Map function requires understanding the AI supply chain. ISO/IEC 42001 includes controls for third-party and customer relationships, requiring organizations to ensure suppliers align with the organization's responsible AI approach. The EU AI Act distinguishes providers from deployers, imposes specific deployer obligations such as human oversight and monitoring, and includes general-purpose AI model provisions that affect downstream users. ISACA's own guidance, including the Digital Trust Ecosystem Framework and COBIT, emphasizes that accountability for enterprise outcomes always resides with the enterprise and its governance body, with vendor management as an enabling process. ISO/IEC 23894 extends ISO 31000 risk management principles to AI, reinforcing that risk ownership is internal. The exam expects you to recognize that these frameworks converge on one message: accountability stays with the deploying organization.

6. Exam Tips: Answering Questions on Accountability for Third-Party and Embedded AI

Tip 1: Accountability never transfers to the vendor. If an answer option suggests that the organization has discharged its duty because the vendor is responsible, certified, or contractually liable, it is almost always wrong. The correct answer will keep accountability with the deploying organization while using contracts and assurance to manage the vendor relationship.

Tip 2: Look for the BEST or FIRST action. ISACA questions frequently ask what should be done FIRST. For third-party AI, the first step is usually to identify and inventory the AI or to perform a risk assessment, not to sign a contract, deploy controls, or monitor. Governance precedes controls; controls precede monitoring.

Tip 3: Distinguish accountability, responsibility, and liability. The exam may test whether you know the difference. Accountability belongs to a single internal role answerable for outcomes. Responsibility can be shared or delegated to those performing work, including vendors. Liability is a legal consequence that contracts can allocate. Only one party should be accountable; several can be responsible.

Tip 4: Embedded AI questions test discovery. When a scenario describes a vendor adding AI functionality to an existing product, the risk being tested is that the AI entered the environment without assessment. The best answer typically involves updating the AI inventory, assessing the new feature against the risk tiering process, and deciding whether to enable, restrict, or disable it before use.

Tip 5: Prefer evidence over assurances. When options compare vendor statements, marketing claims, or self-attestations against independent audits, test results, model documentation, or contractual rights to verify, choose evidence and verification rights.

Tip 6: Think about the intended use boundary. Many scenarios involve a system approved for one purpose being used for another. The correct response recognizes that validation applies only to the assessed context and that a new use case requires a new risk assessment.

Tip 7: Identify the right internal owner. If asked who should be accountable for a third-party AI system used in a business process, the answer is the business owner of that process or the relevant senior executive, not the IT department, not the procurement team, and not the vendor. Oversight functions such as risk and compliance support and challenge, but do not own the business outcome.

Tip 8: Recognize model change and drift as ongoing risks. Scenarios describing a vendor updating its model, or performance degrading over time, test continuous monitoring and change notification clauses. The best answer involves reassessment triggered by material change, not a one-time acceptance at procurement.

Tip 9: Remember fourth-party and concentration risk. If a scenario mentions that a vendor relies on an upstream foundation model provider, or that many systems depend on one provider, the risk being tested is supply-chain or concentration risk. Appropriate responses include supply-chain transparency requirements, contingency planning, and diversification where feasible.

Tip 10: Map to the governance layer, not just the technical layer. Because this topic sits in the governance domain, the strongest answers reference policy, roles, committees, risk appetite, inventory, and framework alignment. Purely technical answers, such as adding a filter or encrypting data, may be valid controls but are rarely the BEST governance response.

Tip 11: Eliminate extreme options. Answers that say to ban all third-party AI, or to trust all certified vendors without review, are rarely correct. ISACA favors risk-based, proportionate approaches tied to risk tiering.

Tip 12: Watch for data flow questions. If a scenario involves sending personal, confidential, or regulated data to a vendor AI, the issue is usually data governance: lawful basis, consent, contractual restrictions on vendor training, and data residency. The deploying organization remains the data controller and remains accountable.

7. Sample Question Walkthrough

An organization discovers that its human resources platform vendor has enabled an AI-based candidate ranking feature in a recent update. Recruiters have begun using it. Which of the following should the AI risk manager do FIRST?

A. Negotiate contractual indemnification with the vendor
B. Add the feature to the AI inventory and perform a risk assessment of its use
C. Implement bias monitoring on ranking outputs
D. Train recruiters on responsible use of the feature

Correct answer: B. The feature is embedded AI that entered the environment without assessment. Before contracts, controls, or training can be properly scoped, the organization must recognize the system exists and understand its risk, which for candidate ranking is likely high. Options A, C, and D are all legitimate subsequent actions, but they depend on the outcome of the assessment. This illustrates the exam logic: discover and assess first, then govern, then control, then monitor.

8. Key Takeaways

• Accountability for AI outcomes remains with the deploying organization regardless of who built the system
• Embedded AI is the hidden risk; discovery and inventory are the foundational controls
• Shared responsibility must be made explicit through documentation and contracts
• Due diligence for AI goes beyond security questionnaires to include training data, fairness, transparency, and change management
• A named internal business owner must be accountable for each system
• Monitoring and reassessment are continuous because vendor models change
• Frameworks such as NIST AI RMF, ISO/IEC 42001, and the EU AI Act all reinforce deployer accountability
• On the exam, choose risk-based, evidence-driven, governance-first answers that keep accountability inside the organization