Ongoing Vendor Monitoring and Attestations

Why Ongoing Vendor Monitoring and Attestations Matter

Most organizations do not build their AI systems from scratch. They license foundation models through APIs, embed AI features from SaaS providers, rely on cloud platforms for training and inference, and pull in third-party datasets, labeling services, and open-source components. Each of these relationships transfers operational work to the vendor but never transfers accountability. Regulators, customers, and boards hold the deploying organization responsible for the outcomes of the AI it uses, regardless of who built it.

This creates a unique risk problem. Traditional third-party risk management often treated due diligence as a one-time gate at onboarding. AI vendors, however, change constantly: models are retrained, versions are deprecated, training data sources shift, subprocessors are added, pricing and terms change, and new vulnerabilities (such as prompt injection or data leakage paths) are discovered. A vendor that passed assessment twelve months ago may present a materially different risk profile today. Ongoing monitoring closes the gap between point-in-time assurance and the continuous reality of AI supply chains, while attestations provide structured, independent or self-declared evidence that controls remain in place.

For the ISACA AAIR exam, this topic sits within the AI Risk Program Management domain. The exam expects you to understand that an AI risk program is only as strong as its weakest supplier, that monitoring must be risk-based and proportionate, and that attestations are inputs to risk decisions rather than substitutes for judgment.

What It Is: Core Definitions

Ongoing vendor monitoring is the continuous or periodic process of observing, measuring, and evaluating a third party's performance, security posture, compliance status, financial health, and AI-specific risk indicators throughout the life of the relationship, from contract signature to offboarding.

Attestations are formal statements, provided by the vendor or by an independent party, asserting that specified controls, standards, or obligations have been met. Common forms include:
- SOC 1 / SOC 2 Type I and Type II reports (independent auditor opinions on controls; Type II covers operating effectiveness over a period).
- ISO/IEC 27001 (information security management) and ISO/IEC 42001 (AI management system) certifications.
- ISO/IEC 27701 (privacy) and ISO/IEC 23894 alignment statements (AI risk management guidance).
- NIST AI RMF or NIST CSF self-assessments and profiles.
- Vendor self-attestations and questionnaires (e.g., SIG, CAIQ, or custom AI questionnaires).
- Model cards, system cards, and data sheets describing training data, intended use, limitations, and evaluation results.
- Penetration test summaries, red-team reports, and bias or fairness audit results.
- Compliance attestations such as EU AI Act conformity declarations, GDPR Data Processing Agreements, or sector-specific certifications.

A critical distinction the exam tests: a self-attestation is a vendor's own claim, while an independent attestation (such as a SOC 2 Type II) carries greater assurance because a qualified third party has tested the controls. Neither eliminates the need for the deploying organization's own review.

Why AI Vendors Are Different

AI introduces risk dimensions that classic vendor programs often miss:

1. Model drift and silent updates: A provider may retrain or swap models without notice, changing accuracy, bias characteristics, or behavior. Outputs that were acceptable last quarter may no longer be.
2. Data usage and retention: Whether the vendor uses your prompts, documents, or customer data to train or fine-tune models is a central privacy and IP concern that must be contractually locked and monitored.
3. Subprocessor and fourth-party exposure: An AI SaaS vendor may itself call a foundation model provider, which in turn runs on a hyperscaler. Risk concentrates in a few upstream providers.
4. Explainability and transparency: Vendors may be unable or unwilling to disclose training data provenance, limiting your ability to assess copyright, bias, or regulatory exposure.
5. Emerging regulation: Obligations under the EU AI Act, state-level AI laws, and sector regulators evolve rapidly, so vendor compliance posture must be revalidated over time.
6. Novel attack surfaces: Prompt injection, jailbreaking, model inversion, data poisoning, and supply-chain compromise of model weights require AI-specific security attestations.
7. Intended-use boundaries: A model attested as safe for one use case may be high risk in another. Monitoring must confirm the vendor's product is still used within approved scope.

How It Works: The Monitoring and Attestation Lifecycle

Step 1: Inventory and Risk Tiering
Maintain a complete inventory of AI vendors and AI-enabled services, including shadow AI discovered through expense reports, network logs, or employee surveys. Assign each vendor an inherent risk tier based on factors such as data sensitivity, criticality to operations, degree of autonomy of the AI, impact on individuals (e.g., hiring, credit, healthcare decisions), regulatory classification, and substitutability. Tiering drives the frequency and depth of monitoring: high-tier vendors might require quarterly reviews and annual independent attestations; low-tier vendors might need only annual self-attestations and automated monitoring.

Step 2: Establish the Baseline and Contractual Hooks
Monitoring is only effective if the contract creates rights and obligations. Key clauses include: right to audit or right to receive attestation reports, notification of material model changes, data use and retention restrictions, subprocessor disclosure and approval, incident notification timelines, performance SLAs and AI-specific metrics (accuracy, latency, availability), compliance warranties, exit and data return provisions, and liability allocation. The onboarding assessment establishes the baseline against which future changes are measured.

Step 3: Define Monitoring Indicators
Effective programs track both Key Performance Indicators (KPIs) and Key Risk Indicators (KRIs):
- Service availability and incident counts.
- Model performance metrics versus agreed thresholds (accuracy, false positive/negative rates, fairness metrics across protected groups).
- Frequency and nature of model version changes.
- Security findings, CVEs affecting vendor components, and breach disclosures.
- Regulatory actions, litigation, or adverse media involving the vendor.
- Financial health indicators and ownership changes (mergers often alter risk).
- Attestation currency (is the SOC 2 report expired? did the ISO certificate lapse?).
- Changes in subprocessors or data residency.
- Complaint volumes and user feedback linked to AI outputs.

Step 4: Collect and Review Attestations
On a scheduled basis aligned with the risk tier, request current attestations. Reviewing them means more than filing them. A competent reviewer checks:
- Scope: Does the report cover the specific product, region, and infrastructure you use? A SOC 2 covering a vendor's corporate IT but not its model-hosting environment is of limited value.
- Period: Is the coverage window recent and continuous with the prior report (bridge letters fill gaps)?
- Exceptions and qualified opinions: Were control deviations noted, and what is the vendor's remediation status?
- Complementary User Entity Controls (CUECs): Which controls does the vendor assume you perform? These must be mapped to your own controls.
- Subservice organizations: Were fourth parties carved out or included?
- AI-specific content: Does the attestation address bias testing, data governance, model change management, and human oversight, or only generic security?

Step 5: Continuous and Event-Driven Monitoring
Supplement periodic reviews with always-on signals: external security ratings services, threat intelligence feeds, news and regulatory alerts, API telemetry showing output quality or anomaly rates, and internal incident reports tagged to the vendor. Define trigger events that force an out-of-cycle reassessment: a breach notification, a major model version release, a change of ownership, a new regulatory classification, a failed internal test of model outputs, or the expiry of a key attestation.

Step 6: Evaluate, Escalate, and Remediate
Monitoring outputs feed the organization's risk register. When indicators breach thresholds, the program should apply a defined escalation path: engage the vendor for explanation and corrective action plans, apply compensating controls internally (e.g., additional human review of outputs, output filtering, reduced scope of use), adjust the risk rating, escalate to the AI governance committee or risk committee, and, where necessary, suspend use or invoke exit provisions. Decisions and rationale must be documented to demonstrate due care.

Step 7: Report and Govern
Aggregate vendor risk posture should be reported to senior management and the board at a frequency aligned with the enterprise risk appetite. Reports typically show tier distribution, attestation currency rates, open findings by severity, concentration risk in key providers, and trend lines. Governance committees use this to approve risk acceptances and resource remediation.

Step 8: Offboarding
Monitoring continues until the relationship ends. Offboarding verifies data deletion or return (including any fine-tuned model artifacts and embeddings derived from your data), revocation of access and API keys, and confirmation of destruction certificates. Lessons learned update the risk tiering model and questionnaires.

Roles and Responsibilities

- Business owner / relationship owner: Accountable for the vendor relationship and first-line monitoring of performance and usage within scope.
- AI risk or third-party risk management function (second line): Designs the framework, sets tiering criteria, reviews attestations, challenges first-line assessments, maintains the register.
- Legal and procurement: Negotiate contractual rights and manage renewals.
- Information security and privacy: Evaluate technical and data-protection attestations.
- Internal audit (third line): Independently assesses whether the monitoring program operates as designed.
- AI governance committee / board: Sets risk appetite, approves high-risk vendor relationships, and receives reporting.

Common Weaknesses the Exam Likes to Probe

- Treating a received SOC 2 report as proof of control without reading scope, exceptions, or CUECs.
- Monitoring only at renewal rather than continuously.
- Failing to extend monitoring to fourth parties and foundation-model dependencies.
- Relying solely on vendor self-attestation for high-risk AI use cases.
- Lacking contractual notification rights for model changes.
- Not linking monitoring results to the enterprise risk register and governance decisions.
- Ignoring shadow AI tools adopted without procurement involvement.
- Using the same monitoring intensity for all vendors regardless of risk.

Exam Tips: Answering Questions on Ongoing Vendor Monitoring and Attestations

Tip 1: Accountability never transfers. If an answer option implies that outsourcing AI to a vendor removes the organization's responsibility, it is wrong. The deploying organization remains accountable; the vendor is responsible for its contracted obligations.

Tip 2: Prefer risk-based and proportionate approaches. When options range from 'monitor all vendors identically' to 'tier vendors by inherent risk and scale monitoring accordingly,' the risk-based answer is almost always correct. ISACA consistently favors proportionality over blanket controls.

Tip 3: Independent beats self-declared, but neither replaces review. If asked which provides the highest assurance, choose independent attestations such as a SOC 2 Type II or ISO 42001 certification over questionnaires. If asked what the organization should do upon receiving a report, choose 'review scope, period, exceptions, and complementary user entity controls' over 'file the report as evidence of compliance.'

Tip 4: Type II over Type I for ongoing assurance. Type I reports describe control design at a point in time; Type II tests operating effectiveness over a period. For ongoing monitoring questions, Type II is the stronger evidence.

Tip 5: Look for the AI-specific angle. Many distractors are generic IT vendor management answers. If the question mentions model retraining, data used for training, bias, or model versioning, the best answer will address model change notification, data-use restrictions, fairness testing, or model cards rather than only encryption or uptime.

Tip 6: Contract first, then monitor. If a scenario describes an inability to obtain information from a vendor, the root cause is often a missing contractual right. The best preventive answer is to include audit, attestation, and notification clauses before onboarding; the best detective answer is to request the attestation and escalate gaps.

Tip 7: Triggers require out-of-cycle action. When a scenario introduces a breach, acquisition, major model update, or regulatory change at the vendor, the correct response is a trigger-based reassessment, not waiting for the scheduled annual review.

Tip 8: Fourth parties count. If a question highlights that the vendor relies on a foundation model provider or cloud host, the correct answer usually involves identifying and assessing subprocessors and concentration risk, not assuming the vendor's attestation covers them automatically.

Tip 9: Expired attestation equals a gap, not a failure to be ignored. If a SOC report has lapsed, the appropriate step is to request a bridge letter or updated report and treat the gap as a risk pending resolution, potentially with compensating controls, rather than assuming controls still operate.

Tip 10: Connect monitoring to governance. Answers that end at 'the analyst noted the finding' are incomplete. The best answers close the loop: update the risk register, escalate per defined thresholds, decide on remediation, acceptance, or exit, and report to governance bodies.

Tip 11: Read for the keyword 'BEST,' 'FIRST,' or 'MOST important.' For 'FIRST,' choose foundational steps such as inventorying AI vendors or defining risk tiers. For 'MOST important' in attestation review, choose scope and applicability to your usage. For 'BEST' ongoing assurance, choose continuous monitoring combined with periodic independent attestation.

Tip 12: Eliminate extremes. Options such as 'terminate the vendor immediately' or 'accept the vendor's statement without further action' are rarely correct unless the scenario describes a severe, unremediated, intolerable risk or a trivially low-risk relationship.

Sample Question Walkthrough

An organization uses a third-party generative AI service for customer support. The vendor's most recent SOC 2 Type II report covers its corporate network but excludes the model-hosting platform operated by a subprocessor. What should the AI risk manager do FIRST?

A. Accept the report because SOC 2 Type II is the highest standard of assurance.
B. Terminate the contract due to insufficient coverage.
C. Identify the subprocessor and obtain or request attestation covering the model-hosting environment.
D. Increase the internal budget for customer support quality review.

Reasoning: Option A ignores scope, a classic trap. Option B is an extreme. Option D is a compensating control that may follow but does not address the information gap. Option C correctly recognizes the fourth-party coverage gap and takes the proportionate first step. Answer: C.

Key Takeaways

- Ongoing vendor monitoring converts point-in-time due diligence into continuous assurance across the AI supply chain.
- Attestations are evidence to be evaluated for scope, period, exceptions, and relevance, not trophies to be collected.
- AI vendors require monitoring of model changes, data use, bias, subprocessors, and regulatory posture in addition to traditional security and operational metrics.
- Risk tiering drives frequency and depth; contractual rights enable monitoring; trigger events force reassessment; governance closes the loop.
- On the exam, choose answers that are risk-based, accountability-preserving, AI-specific, contractually grounded, and connected to governance decision-making.