Model Cards, Datasheets and AI Documentation

1. Why This Topic Matters

AI systems are only as trustworthy as the evidence that supports them. In traditional IT, security managers rely on configuration baselines, change records, asset inventories and system documentation to assert control. In AI, the equivalent evidence is AI documentation: structured records that describe where a dataset came from, how a model was built, what it was tested against, where it should and should not be used, and what risks remain. Without this documentation an organisation cannot demonstrate accountability, cannot perform meaningful risk assessment, cannot audit the system, and cannot respond credibly to regulators, customers or incident investigators.

For the AAISM candidate, this topic sits in the AI Technologies and Controls domain because documentation is itself a control. It is a preventive control (it forces teams to think through intended use, limitations and risks before release), a detective control (it exposes gaps, bias and data quality problems during review) and a corrective and compliance control (it supports incident response, root-cause analysis and regulatory reporting). Regulators and standards bodies have converged on this view: the EU AI Act requires technical documentation for high-risk systems, ISO/IEC 42001 requires documented information across the AI management system, and the NIST AI RMF embeds documentation throughout its Map, Measure and Manage functions. If you remember one idea from this guide, remember this: documentation turns an opaque model into a governable asset.

2. What It Is: The Core Artefacts

Model Cards
Introduced by Mitchell et al. (Google, 2019), a model card is a short, standardised document that accompanies a trained machine learning model. It is written for a mixed audience – developers, risk managers, auditors, regulators and sometimes end users – and typically contains:
• Model details: owner, version, date, model type and architecture, training algorithm, licence, point of contact.
• Intended use: primary intended uses and users, and explicitly out-of-scope uses.
• Factors: relevant demographic, environmental or technical factors that could affect performance.
• Metrics: which performance measures were used and why (accuracy, precision, recall, F1, AUC, calibration, fairness metrics) and decision thresholds.
• Evaluation data: the datasets used for testing, how they were chosen and preprocessed.
• Training data: a summary of what the model was trained on (often a pointer to a datasheet).
• Quantitative analyses: disaggregated results across subgroups and intersectional groups.
• Ethical considerations: sensitive data, potential harms, mitigations.
• Caveats and recommendations: known limitations, conditions under which performance degrades, monitoring advice.

Datasheets for Datasets
Proposed by Gebru et al. (2018), a datasheet is modelled on the component datasheets used in electronics engineering. It documents a dataset rather than a model and is organised around the data lifecycle:
• Motivation: why the dataset was created, by whom, and who funded it.
• Composition: what the instances represent, how many there are, whether it is a sample of a larger population, labels, missing data, confidential or sensitive content, and whether individuals can be identified.
• Collection process: how data was acquired, by whom, over what time frame, consent obtained, ethical review.
• Preprocessing, cleaning and labelling: what was done, whether the raw data is retained, and what tooling was used.
• Uses: previous uses, recommended uses, uses that should be avoided.
• Distribution: how it is shared, licensing, export controls, and any restrictions.
• Maintenance: who maintains it, how errors are reported, update and retention policy.

Related Documentation Formats
• System Cards (used by OpenAI, Meta and others): describe an entire deployed system – model plus guardrails, filters, retrieval components, user interface and safety evaluations – rather than a single model.
• FactSheets (IBM): supplier-declared conformity statements covering purpose, performance, safety, security and lineage, designed to be checked at each lifecycle stage.
• Data Statements (Bender and Friedman): specifically for natural language datasets, covering speaker demographics, language variety and annotator characteristics.
• Dataset Nutrition Labels: a summary label highlighting key data quality and provenance facts.
• Transparency Notes and Responsible AI Impact Assessments: vendor-facing and internal documents explaining capabilities, limitations and intended deployment contexts.
• AI Bill of Materials (AIBOM) and ML-BOM: an inventory of models, datasets, libraries, weights and dependencies that make up an AI system, extending the software bill of materials concept to AI supply-chain security.
• Technical documentation under the EU AI Act (Annex IV): a legally mandated dossier for high-risk systems covering general description, design and development process, monitoring, risk management, and changes over the lifecycle.

3. How It Works: Documentation as a Lifecycle Control

Where it fits in the AI lifecycle
Documentation is not a single deliverable produced at the end; it accumulates across the lifecycle:
1. Problem framing and design: record intended purpose, success criteria, affected stakeholders, regulatory classification and initial risk assessment.
2. Data acquisition: create or obtain datasheets; record provenance, consent, lawful basis, licensing and any synthetic or scraped sources.
3. Development and training: capture model architecture, hyperparameters, training environment, random seeds, version of code and data (lineage), and experiment tracking logs.
4. Evaluation and validation: document test datasets, metrics, disaggregated performance, robustness tests, adversarial and red-team findings, and residual risks.
5. Deployment: publish the model card or system card; record deployment configuration, guardrails, monitoring thresholds and human oversight arrangements.
6. Operation and monitoring: log drift, incidents, user feedback, retraining events and model version changes; update documentation with each change.
7. Retirement: record decommissioning rationale, data disposal and retention of documentation for audit and legal purposes.

Roles and responsibilities
• Data scientists and ML engineers author the technical content.
• Data owners and stewards own datasheets and provenance records.
• Product or business owners define intended use and out-of-scope use.
• Risk, privacy, legal and security functions review and approve, and map the documentation to regulatory obligations.
• Internal audit uses documentation as evidence of control operation.
• Senior management and the AI governance committee are accountable for ensuring documentation exists, is current and is used in decision-making.

What good documentation enables
• Transparency and explainability: stakeholders understand what the system does and does not do.
• Accountability: named owners, approvals and decision trails.
• Risk assessment and bias evaluation: disaggregated metrics expose disparate performance.
• Third-party and supply-chain assurance: procurement teams can evaluate vendor models and foundation models before adoption; missing or vague documentation is itself a risk indicator.
• Change and configuration management: versioned cards show what changed and why.
• Incident response: responders know the training data, dependencies and known weaknesses.
• Regulatory compliance: evidence for EU AI Act conformity, ISO/IEC 42001 certification, NIST AI RMF alignment, GDPR Article 35 DPIAs and sector regulators.
• Reproducibility: lineage records allow models to be rebuilt or independently validated.

Security-specific considerations
Documentation is also a security artefact. Model cards and datasheets should record: threat model assumptions; adversarial robustness testing; data poisoning controls applied during collection; prompt injection and jailbreak testing for LLM systems; access controls on weights and training data; hashing or signing of model artefacts for integrity; and the AIBOM of third-party components with known vulnerabilities. At the same time, documentation can leak sensitive information – detailed descriptions of training data, architectures or weaknesses may aid attackers or expose personal data. Organisations therefore commonly maintain a tiered approach: a public card with high-level information, and an internal, access-controlled version with full technical and security detail.

Common weaknesses auditors find
• Documentation created once and never updated after retraining.
• Intended use stated but out-of-scope use omitted.
• Aggregate accuracy reported with no subgroup breakdown.
• No provenance or licensing information for scraped or purchased data.
• Vendor or open-source models adopted with no card at all.
• Documentation stored informally (wiki pages, notebooks) outside version control and without approval workflow.
• No link between documentation and the risk register or control framework.

4. Exam Tips: Answering Questions on Model Cards, Datasheets and AI Documentation

Tip 1 – Know which artefact documents what. A datasheet documents a dataset (motivation, composition, collection, preprocessing, uses, distribution, maintenance). A model card documents a trained model (intended use, metrics, evaluation data, disaggregated results, limitations). A system card documents the whole deployed system including guardrails. An AIBOM documents components and dependencies. If a question asks where to record data consent or sampling methodology, the answer is the datasheet, not the model card.

Tip 2 – Think like a manager, not a data scientist. AAISM is a management-level certification. When options include a highly technical action (tune hyperparameters) and a governance action (require a model card with out-of-scope uses before approval), the governance action is usually correct. The exam rewards answers that establish accountability, evidence and oversight.

Tip 3 – Documentation is a control, so apply control logic. Expect scenario questions where the best answer is the one that produces evidence, assigns ownership, or integrates documentation into an existing process (change management, procurement, risk register). Documentation that is not reviewed, approved, versioned and updated is a weak control – questions often test whether you recognise that stale documentation is a finding.

Tip 4 – Remember the primary purpose: transparency and accountability. If asked why model cards are used, the strongest answer centres on communicating intended use, limitations and performance across subgroups to enable informed, accountable decisions – not on improving model accuracy or reducing training cost.

Tip 5 – Link to frameworks. Be ready to associate documentation requirements with the EU AI Act (technical documentation for high-risk systems, transparency obligations), ISO/IEC 42001 (documented information, AI system impact assessment, supplier controls), NIST AI RMF (Map, Measure, Manage functions and the Govern function cross-cutting), and privacy law (records of processing, DPIAs). You do not need article numbers, but you should recognise that documentation is a cross-framework expectation.

Tip 6 – Third-party and foundation model scenarios. When a vendor cannot or will not supply a model card, datasheet or AIBOM, the correct response is to treat this as a risk: escalate, require contractual documentation obligations, perform independent testing, or restrict use – never to assume the vendor has handled it. Reviewing vendor documentation is a due-diligence control, and its absence is a red flag.

Tip 7 – Disaggregated metrics signal bias awareness. Questions on fairness often hinge on whether performance was reported by subgroup. An answer that mentions only overall accuracy is incomplete; the better answer requires performance broken down by relevant factors (demographics, environments, input conditions) and documents residual disparities.

Tip 8 – Balance transparency and confidentiality. If a question raises the risk that publishing detailed documentation could expose trade secrets, personal data or attack vectors, the best answer is a tiered or redacted disclosure approach with full detail retained internally under access control – not withholding documentation entirely, and not publishing everything.

Tip 9 – Lifecycle wording matters. Watch for phrases such as after retraining, new deployment context or data source changed. These signal that documentation must be updated and re-approved. Choose the option that triggers a review of the model card and datasheet rather than one that assumes prior documentation remains valid.

Tip 10 – Eliminate distractors. Wrong answers often: confuse a model card with a privacy notice or end-user licence; suggest documentation replaces testing or monitoring (it complements them); propose that documentation alone proves compliance (it is evidence, not assurance); or recommend documenting only successful results. The correct answer typically presents documentation as honest, complete, versioned, owned and integrated with risk management.

Quick Recall Summary
• Model card = model; datasheet = dataset; system card = full system; AIBOM = components.
• Core purpose: transparency, accountability, informed use, auditability.
• Must include intended use and out-of-scope use; metrics and disaggregated results; limitations and mitigations.
• Living documents: update on retraining, redeployment or data change.
• Governance expectation under EU AI Act, ISO/IEC 42001 and NIST AI RMF.
• Missing vendor documentation = supply-chain risk to be escalated.
• Tiered disclosure balances openness with security and confidentiality.
• In the exam, favour the answer that creates evidence, assigns ownership and embeds documentation into existing control processes.