AI Conformity Assessment and Recordkeeping

Domain Context: AI Risk Governance and Framework Integration (ISACA Advanced in AI Risk – AAIR)

1. Why This Topic Matters

Conformity assessment and recordkeeping are the mechanisms that turn AI governance from a set of good intentions into demonstrable, auditable evidence. Regulators, auditors, customers and boards increasingly ask one question: Can you prove that your AI system meets the requirements that apply to it? If the answer depends on memory, tribal knowledge or scattered emails, the organization has a governance gap and a risk exposure.

Key reasons this topic is heavily weighted in AI risk programs and on the AAIR exam:

• Regulatory enforcement is real. The EU AI Act requires high-risk AI systems to undergo a conformity assessment before being placed on the market or put into service, and imposes fines of up to 7% of global annual turnover or EUR 35 million for the most serious breaches. Other jurisdictions (Canada, Brazil, South Korea, several US states) are adopting similar assurance and documentation expectations.
• Accountability requires evidence. Frameworks such as the NIST AI Risk Management Framework (Govern, Map, Measure, Manage), ISO/IEC 42001 (AI Management System) and ISO/IEC 23894 (AI risk management) all depend on documented information to prove that controls exist and operate.
• Auditability and incident response. When an AI system causes harm, logs and technical documentation are what allow investigators to reconstruct what happened, which model version was running, what data it used and who approved it.
• Framework integration. AI conformity activities cannot live in a silo; they must integrate with existing quality (ISO 9001), information security (ISO/IEC 27001), privacy (ISO/IEC 27701, GDPR) and enterprise GRC processes. The AAIR exam tests whether you can connect these pieces.

2. What Is AI Conformity Assessment?

Definition: Conformity assessment is the process of demonstrating that an AI system, and the processes around it, fulfil specified requirements – legal, regulatory, contractual, standards-based or internal policy requirements. The ISO/IEC 17000 family defines conformity assessment broadly as a demonstration that specified requirements are fulfilled.

Types of conformity assessment (know these distinctions cold):

• First-party (internal / self-assessment): Performed by the provider itself, based on internal control. Under the EU AI Act this is the Annex VI route, available for most high-risk systems in Annex III (e.g., employment, education, credit scoring) provided harmonised standards are applied.
• Second-party: Performed by a party with a user interest, such as a customer or deployer assessing a vendor's AI system (vendor due diligence, contractual audits).
• Third-party (independent): Performed by an independent body – under the EU AI Act, a notified body designated by a national authority (Annex VII route). Mandatory for certain categories, notably remote biometric identification systems where harmonised standards are not fully applied, and for AI that is a safety component of products already subject to third-party assessment (e.g., medical devices, machinery).

Core outputs of an EU AI Act-style conformity assessment:

• Technical documentation (Annex IV) demonstrating compliance with requirements for risk management, data governance, transparency, human oversight, accuracy, robustness and cybersecurity.
• EU Declaration of Conformity – a written, signed statement by the provider that the system meets the applicable requirements; the provider assumes legal responsibility by signing it.
• CE marking – affixed to indicate conformity (physical or digital).
• Registration of the high-risk system in the EU database before placing on the market.
• Quality management system (Article 17) covering strategy, design controls, testing, data management, post-market monitoring, incident reporting and accountability.

Triggers for re-assessment: A substantial modification – a change not foreseen in the original assessment that affects compliance or intended purpose – requires a new conformity assessment. For continuously learning systems, changes that were pre-determined and documented in the original assessment do not constitute a substantial modification. This nuance is a classic exam point.

3. What Is AI Recordkeeping?

Recordkeeping is the systematic creation, retention, protection and retrieval of records that evidence how an AI system was designed, tested, approved, deployed, monitored and changed. It operates on two levels:

A. System-level automatic logging (traceability)
• The EU AI Act (Article 12) requires high-risk systems to be technically capable of automatically recording events (logs) over their lifetime.
• Logs must enable identification of situations that may cause risk or substantial modification, facilitate post-market monitoring and support deployer monitoring of operation.
• For biometric systems, logs must include the period of each use, the reference database checked, input data that led to a match, and the identity of persons involved in verification.
• Retention: Providers and deployers must keep automatically generated logs for a period appropriate to the intended purpose, and at least six months, unless other law provides otherwise.

B. Organizational and lifecycle documentation
• Technical documentation kept by the provider for 10 years after the system is placed on the market or put into service (Article 18), together with the QMS documentation, the Declaration of Conformity, notified body decisions and change approvals.
• AI inventory / register – a catalogue of all AI systems, their risk classification, owner, intended purpose, data sources and status.
• Risk register and risk assessment records – identified risks, likelihood/impact, treatment decisions, residual risk acceptance by accountable owners.
• Data governance records – data provenance, lineage, quality checks, bias examinations, consent and licensing.
• Model documentation – model cards, datasheets for datasets, system cards, validation and test reports, performance and fairness metrics, known limitations.
• Human oversight and decision records – who can override, how overrides are logged, escalation paths.
• Change management and version control – model versions, retraining events, prompt or configuration changes, approvals.
• Incident and serious-incident reports – including regulatory notifications (EU AI Act requires reporting serious incidents to market surveillance authorities, generally within 15 days, shorter for widespread or fatal incidents).
• Post-market monitoring plan and results.
• Training and competence records for staff involved in AI development and oversight (AI literacy obligations).

Under ISO/IEC 42001, these are collectively documented information – the management system must control their creation, update, access, versioning and retention, exactly as ISO/IEC 27001 does for information security.

4. How It Works: The End-to-End Process

Step 1 – Classify the AI system. Determine whether it is prohibited, high-risk, limited-risk (transparency obligations) or minimal-risk under applicable law, and assign an internal risk tier. Classification drives which conformity route and which records are required. Record the classification rationale – this itself is a key record.

Step 2 – Identify applicable requirements. Map legal obligations (EU AI Act, sector regulations, privacy law), standards (ISO/IEC 42001, 23894, 24028, NIST AI RMF, harmonised standards), contractual commitments and internal policies into a requirements register or control library.

Step 3 – Establish the governance and QMS foundation. Define roles (AI system owner, risk owner, compliance, legal, data protection officer, internal audit), the three lines model, decision rights and escalation. Integrate AI controls into the existing enterprise GRC platform rather than building a parallel system.

Step 4 – Design and build with evidence capture. Embed documentation into the lifecycle: data provenance captured at ingestion, experiment tracking, model registries, automated test reports, bias and robustness evaluations, security testing, human oversight design. Evidence generated as a by-product of engineering is more reliable and cheaper than evidence reconstructed later.

Step 5 – Perform the conformity assessment. Select the route (internal control vs. notified body). Compile the technical documentation, test against requirements, remediate gaps, obtain sign-offs. For third-party routes, submit to the notified body and respond to findings. Issue the Declaration of Conformity, apply CE marking, register the system.

Step 6 – Deploy with logging enabled. Confirm automatic logging is active, log integrity is protected (tamper-evident, access-controlled), and retention is configured. Deployers must keep logs under their control and use the system per the instructions for use.

Step 7 – Monitor, report and maintain records. Run the post-market monitoring plan, track drift and incidents, report serious incidents, review human oversight effectiveness. Feed results back into the risk register.

Step 8 – Manage change and re-assess. Evaluate every change for substantial-modification status. Maintain version history. Re-run conformity assessment when triggered. Periodically review the entire program through internal audit and management review.

Step 9 – Retain, protect and retire. Apply retention schedules (e.g., 10 years for technical documentation, minimum six months for logs), protect records against alteration and loss, and ensure retrievability for regulators and auditors. Document decommissioning decisions.

Roles and responsibilities to remember:
• Provider – develops or places on the market; owns conformity assessment, technical documentation, declaration, CE marking, registration, post-market monitoring.
• Deployer – uses the system professionally; must follow instructions, ensure human oversight, keep logs, conduct fundamental rights impact assessments where required, inform affected persons, and monitor operation.
• Importer / Distributor – verify conformity evidence exists before making available.
• Notified body – independent third-party assessor.
• Market surveillance authority – enforces, can request documentation and logs.
• Board / senior management – ultimately accountable for AI risk governance; approve risk appetite and resourcing.
• Internal audit (third line) – provides independent assurance that conformity and recordkeeping controls operate.

5. Framework Integration: How Conformity and Records Connect to Other Standards

• NIST AI RMF: GOVERN requires policies and accountability structures; MAP requires documented context and intended purpose; MEASURE requires recorded test results and metrics; MANAGE requires documented risk treatment and monitoring. Conformity assessment is effectively a formalized MEASURE/MANAGE cycle; records are the artifacts of all four functions.
• ISO/IEC 42001: Clause 7.5 (documented information), Clause 9 (performance evaluation, internal audit, management review) and Annex A controls (AI system impact assessment, data for AI systems, AI system life cycle, information for interested parties) map directly to recordkeeping and assessment duties. Certification to 42001 is itself a third-party conformity assessment of the management system (not of an individual AI product).
• ISO/IEC 27001 and SOC 2: Reuse logging, access control, change management and evidence collection processes; extend them to models and training data.
• GDPR: Data protection impact assessments (DPIAs) and records of processing complement AI technical documentation; the EU AI Act fundamental rights impact assessment can build on the DPIA.
• COBIT / ISACA guidance: Governance system components, RACI assignments, performance management and assurance align with maintaining a defensible AI control environment.

6. Common Risks and Control Weaknesses (Frequently Tested)

• Treating conformity assessment as a one-time, pre-launch event instead of a lifecycle obligation.
• Failing to recognize a substantial modification (e.g., retraining on new data that changes behaviour, expanding intended purpose) and therefore not re-assessing.
• Logs that exist but are incomplete, not tamper-evident, not retained long enough, or not retrievable.
• Technical documentation drafted after the fact, inconsistent with the actual deployed version.
• No AI inventory – you cannot assess or document what you do not know you have.
• Over-reliance on vendor assurances without obtaining and reviewing their conformity evidence (second-party assessment gap).
• Shadow AI and generative AI tools deployed outside governance.
• Confusing the provider's and deployer's record obligations.
• Lack of independent review – self-assessment with no internal audit or quality check.

7. How to Answer Exam Questions on This Topic

ISACA exam questions are scenario-based and typically ask for the BEST, MOST important, FIRST, PRIMARY or GREATEST answer. Several options may be partially correct; you must pick the one that best reflects an AI risk professional's priorities. Apply the following reasoning framework:

• Governance before technology. If an option establishes accountability, classification, policy or a risk-based approach, it usually precedes technical fixes.
• Risk-based and proportionate. The level of assessment rigor and documentation should scale with the risk classification of the system, not be uniform for every model.
• Evidence over assertion. Prefer answers that produce verifiable, retained evidence (test reports, signed approvals, logs) over answers that rely on statements, awareness or intent.
• Lifecycle, not point-in-time. Answers that embed continuous monitoring, change triggers and periodic re-assessment are stronger than one-off certifications.
• Independence matters. When the question stresses assurance, objectivity or high-risk use, an independent (third-party or internal audit) review is usually the best answer.
• Know who owns what. Provider versus deployer obligations; management owns risk acceptance, not the risk function; internal audit assures, it does not design controls.
• Integrate, do not duplicate. If an option leverages existing GRC, QMS or ISMS processes to cover AI, it is typically preferred over building separate siloed processes.

8. Exam Tips: Answering Questions on AI Conformity Assessment and Recordkeeping

Tip 1 – Read for the trigger word. If a scenario describes a change to the model, data, or intended purpose, ask yourself whether it is a substantial modification. If yes, the best answer involves re-performing the conformity assessment and updating documentation, not merely notifying users.

Tip 2 – FIRST questions want the foundational step. When asked what to do FIRST in building a conformity program, the answer is usually to inventory and classify AI systems or to identify applicable requirements – not to purchase tooling or draft a declaration.

Tip 3 – MOST important record = the one that proves accountability. Among log files, marketing collateral, user manuals and signed risk-acceptance records, the record demonstrating that an accountable owner reviewed and accepted residual risk is typically the strongest governance evidence.

Tip 4 – Remember key numbers. Technical documentation: 10 years after placing on the market. Automatic logs: at least six months (longer if other law or purpose requires). Serious incident reporting: generally within 15 days, faster for widespread infringement or death. Exam questions may test whether you know that at least implies a minimum, with risk appetite and other laws potentially requiring longer retention.

Tip 5 – Distinguish management system certification from product conformity. ISO/IEC 42001 certification proves the organization has a functioning AI management system; it does not by itself constitute the EU AI Act product conformity assessment for a specific high-risk system. A question offering 42001 certification as proof of product compliance is offering a distractor.

Tip 6 – Self-assessment is acceptable only where permitted. If the scenario involves remote biometric identification or an AI safety component in a regulated product, lean toward third-party (notified body) assessment. For most Annex III use cases with harmonised standards applied, internal control is acceptable – but internal audit should still provide assurance.

Tip 7 – Logs must be usable, not just present. If options include enabling logging versus ensuring logs are complete, protected from tampering, retained and retrievable, the latter is the stronger control answer. Think integrity, availability and traceability.

Tip 8 – Vendor AI still needs your evidence. When an organization deploys a third-party AI system, the best answer usually involves obtaining the provider's conformity documentation, verifying it against your intended use, and maintaining your own deployer records (logs, oversight, impact assessment) – not assuming the vendor's CE marking covers your obligations.

Tip 9 – Documentation must match reality. If a scenario mentions that documentation was prepared before launch and the system has since been updated several times, the risk is documentation drift. The best response is to establish change-controlled, version-linked documentation, often automated through the model registry and CI/CD pipeline.

Tip 10 – Eliminate options that are purely reactive, purely technical or purely procedural. ISACA favors answers that combine governance, risk-based prioritization and demonstrable control operation. An answer such as 'increase model accuracy' or 'provide training' rarely beats 'establish a risk-based conformity assessment process with defined evidence requirements and ownership'.

Tip 11 – Watch for role confusion distractors. Options that assign risk acceptance to the compliance officer, control design to internal audit, or conformity declarations to the deployer are usually wrong because they violate the three lines model or regulatory role definitions.

Tip 12 – When two answers seem right, choose the broader lifecycle control. For example, 'perform a conformity assessment before deployment' versus 'establish a conformity assessment process that is triggered at deployment and upon substantial modification, with periodic review' – the second is the BEST answer because it covers the full lifecycle.

9. Practice Scenarios

Scenario A: An organization retrained its high-risk credit-scoring model on three years of new data, which was not foreseen in the original technical documentation. Model performance improved. What is the MOST appropriate action?
Best answer: Treat the retraining as a potential substantial modification, assess its impact on conformity, update the technical documentation and re-perform the conformity assessment before continued use. Why not others: Improved accuracy does not remove the obligation; simply informing users or updating the model card is insufficient.

Scenario B: A deployer uses a vendor's CE-marked recruitment screening system. Which record is the deployer PRIMARILY responsible for maintaining?
Best answer: Logs generated by the system under its control and evidence of human oversight and monitoring during use (plus a fundamental rights impact assessment where required). Why not others: The technical documentation and declaration of conformity are provider obligations.

Scenario C: An AI risk manager is building a conformity program for an enterprise with dozens of undocumented AI tools. What should be done FIRST?
Best answer: Establish an AI inventory and classify each system by risk. Why not others: Retention schedules, logging standards and notified body engagement all depend on knowing what systems exist and their risk level.

Scenario D: During an internal audit, logs for a high-risk system are found to be stored on a developer's laptop and overwritten every 30 days. What is the GREATEST concern?
Best answer: Loss of traceability and non-compliance with minimum retention and integrity expectations, undermining incident investigation and regulatory defensibility. Why not others: Storage cost or developer convenience are not governance concerns of comparable weight.

10. One-Page Summary

• Conformity assessment = proving an AI system meets requirements; routes are first-party (internal), second-party (customer), third-party (notified body).
• Key outputs = technical documentation, declaration of conformity, CE marking, database registration, QMS.
• Recordkeeping = automatic system logs (min. six months) plus lifecycle documentation (technical file retained 10 years), AI inventory, risk register, model and data documentation, change, incident and oversight records.
• Re-assess on substantial modification; continuous post-market monitoring is mandatory, not optional.
• Integrate with NIST AI RMF, ISO/IEC 42001, ISO/IEC 27001, GDPR and enterprise GRC – one evidence base, many requirements.
• Exam mindset = governance first, risk-based, evidence-driven, lifecycle-oriented, correct role ownership, independence for high-risk assurance.

Master these distinctions and the reasoning framework above, and questions on AI conformity assessment and recordkeeping become some of the most predictable – and scoreable – items on the AAIR exam.