Open-Source Models, Datasets and Component Risk

1. Why This Topic Matters

Almost no organisation builds AI from scratch. Modern AI systems are assembled from pre-trained open-source or open-weight models (downloaded from hubs such as Hugging Face), public or third-party datasets used for training, fine-tuning and evaluation, and a deep stack of software components such as PyTorch, TensorFlow, scikit-learn, LangChain, tokenizers, vector databases, inference servers and thousands of transitive Python packages. Each of these is a link in the AI supply chain, and each can introduce risk the organisation did not create and may not even know it has inherited.

For the AAISM (Advanced in AI Security Management) candidate, this topic sits squarely inside the AI Risk Management for Security domain because:
• It is a third-party / supply-chain risk problem, which traditional vendor risk management programmes were never designed to handle (there is often no vendor, no contract and no SLA for a model downloaded from a public repository).
• Compromise can be silent and persistent – a backdoored model or poisoned dataset behaves normally until a trigger appears.
• It is explicitly addressed in major frameworks the exam draws from: OWASP Top 10 for LLM Applications (Supply Chain Vulnerabilities, Data and Model Poisoning), MITRE ATLAS, the NIST AI Risk Management Framework, ISO/IEC 42001, and software supply-chain standards such as SLSA and SBOM practice.
• Regulators increasingly expect organisations to demonstrate provenance, licensing compliance and accountability for the models and data they deploy (for example the EU AI Act transparency and documentation duties for general-purpose models).

In short: the organisation remains accountable for an AI system's behaviour and legal posture regardless of where its ingredients came from. Open source shifts development effort but not responsibility.

2. What It Is – Key Definitions

Open-source model – A model whose weights, and ideally architecture, training code and training data details, are published under a licence permitting use and modification. In practice many popular models are only open-weight: the weights are downloadable but training data and full code are not disclosed, and the licence may be restrictive (e.g., non-commercial, acceptable-use clauses, field-of-use limits). The exam expects you to recognise that open-weight is not the same as fully open-source, and that reduced transparency means reduced ability to assess provenance and bias.

Open or third-party dataset – Data collected by someone else and used for pre-training, fine-tuning, retrieval (RAG corpora), evaluation or benchmarking. Examples include web-scraped corpora, image datasets, public benchmark sets and community-curated instruction-tuning sets. Datasets carry risk around quality, poisoning, licensing/copyright, personal data, bias and representativeness.

Components – Everything else that makes the system run: ML frameworks, libraries, model-serving software, agent frameworks, plugins and tools, tokenizers, prompt templates, container images, model file formats and the package managers that deliver them. Component risk is largely classic software-supply-chain risk (CVEs, malicious packages, dependency confusion) plus AI-specific twists such as unsafe model serialisation.

AI Bill of Materials (AI-BOM / ML-BOM) – An inventory extending the Software Bill of Materials concept to list models, model versions, datasets, licences, hashes and dependencies for a given AI system. It is the foundation for knowing what you have, which is the first step of risk management.

Model card / dataset card (datasheet) – Standardised documentation describing intended use, training data, evaluation results, limitations, licence and ethical considerations. Their presence and quality are a key input to due diligence; their absence is itself a risk indicator.

3. The Risk Landscape – What Can Go Wrong

A. Model-level risks
• Backdoors / trojaned weights – A model is fine-tuned so that a hidden trigger (a word, pixel pattern or phrase) causes attacker-chosen behaviour, while normal performance is unaffected and standard tests pass.
• Unsafe serialisation – Formats such as Python pickle execute arbitrary code on load. A malicious model file can compromise the host the moment it is deserialised. Safer formats (e.g., safetensors) and scanning mitigate this.
• Provenance and integrity failures – No verified publisher, no signature or hash, model swapped or tampered in transit, typosquatted model names on hubs, or a hub account takeover.
• Licensing and legal exposure – Using a non-commercial or restricted-use model in production, violating attribution or acceptable-use terms, or inheriting liability for copyrighted training content.
• Inherited bias, toxicity and unknown training data – The organisation cannot explain or defend outputs because it does not know what the model learned from.
• Model drift and abandonment – Community models may stop being maintained, leaving unpatched issues with no accountable owner.

B. Dataset-level risks
• Data poisoning – Attackers plant manipulated samples in public datasets (or in web content that will be scraped) to degrade performance or implant backdoors during training or fine-tuning.
• Label and quality issues – Mislabelled or low-quality data produces unreliable models; errors in widely used benchmark sets propagate across the industry.
• Personal data and privacy – Datasets scraped without consent may contain PII, special-category data or children's data, creating GDPR-style obligations and memorisation/leakage risk.
• Copyright and licence conflicts – Terms of the dataset may forbid commercial use, require share-alike, or be unclear; downstream models may inherit these restrictions.
• Bias and representativeness – Historical or demographic skew leads to discriminatory outcomes, a compliance and reputational risk.
• Provenance gaps – Inability to prove where data came from undermines audit, regulatory defence and incident response.

C. Component-level risks
• Known vulnerabilities (CVEs) in ML frameworks, inference servers and web front-ends.
• Malicious or typosquatted packages on PyPI, npm or container registries, and dependency confusion where an internal package name is hijacked publicly.
• Transitive dependencies – deep, poorly understood dependency trees where one abandoned library affects everything.
• Agent tools and plugins – third-party tools that an LLM can invoke extend the attack surface to excessive agency, prompt injection relayed through tools, and data exfiltration.
• Insecure defaults in serving software (open admin interfaces, no auth on model endpoints, verbose error leakage).
• Infrastructure components – GPU drivers, container images and orchestration platforms with their own patching cadence.

4. How It Works – Managing the Risk Across the Lifecycle

The AAISM perspective is a management one: the candidate must know how to govern, assess, treat and monitor these risks, not merely how to run a scanner. Think in terms of the classic risk cycle mapped to NIST AI RMF functions (Govern, Map, Measure, Manage).

Step 1 – Govern: set policy and ownership
• Establish an AI acceptable-use and sourcing policy that defines approved sources (e.g., internal registry, vetted hubs), prohibited sources, and who may introduce external models, data and packages.
• Define a risk owner for each AI system and integrate open-source AI components into the existing third-party risk management (TPRM) and secure development lifecycle rather than creating a parallel process.
• Align decisions with the organisation's risk appetite; some use cases may justify only commercially supported or internally trained models.
• Involve legal/IP counsel for licence review and privacy for dataset review.

Step 2 – Map / Identify: know what you have
• Maintain an AI inventory and generate an AI-BOM for each system listing models, versions, hashes, datasets, licences and dependencies.
• Discover shadow AI: developers pulling models and packages outside sanctioned channels.
• Document intended use and classify systems by criticality and data sensitivity, because this drives the depth of due diligence.

Step 3 – Measure / Assess: perform due diligence
• Provenance verification – confirm publisher identity, signatures, checksums, repository reputation and history; prefer models with attestations.
• Documentation review – evaluate model cards and datasheets for intended use, training data description, known limitations and evaluation results. Missing documentation raises the risk rating.
• Licence review – confirm compatibility with intended commercial use and downstream distribution; record obligations (attribution, copyleft, use restrictions).
• Security scanning – scan model files for unsafe serialisation and embedded code, scan packages and containers for CVEs and malware, check dependency trees.
• Dataset assessment – sample for quality, PII, toxicity and bias; check collection method and consent basis; evaluate poisoning indicators and anomalous samples.
• Independent evaluation – benchmark and red-team the model yourself (robustness, jailbreak resistance, backdoor probing, fairness tests) rather than trusting published scores.
• Score likelihood and impact and record results in the risk register.

Step 4 – Manage / Treat: apply controls proportionate to risk
• Avoid – reject components that fail licence, provenance or security criteria.
• Mitigate – pin versions and hashes; mirror approved artefacts in an internal registry; convert to safe formats; load models in sandboxed, least-privilege environments; apply input/output filtering and guardrails; segment model-serving infrastructure; enforce code review for dependency changes; use SLSA-style build provenance.
• Transfer – where possible use commercially supported distributions with indemnification, or insurance; recognise that transfer is limited for freely downloaded assets.
• Accept – document residual risk with sign-off by the risk owner at an appropriate level.
• Apply data governance controls for datasets: lineage tracking, access control, validation pipelines, deduplication and poisoning detection before training.

Step 5 – Monitor and respond
• Subscribe to vulnerability feeds for frameworks and to hub advisories for models; re-scan on each update.
• Monitor model behaviour in production for drift, anomalous outputs and signs of trigger activation.
• Maintain incident response playbooks for a compromised model or poisoned dataset, including rollback to a known-good version (which requires the hashes and inventory from Step 2).
• Periodically re-assess licences and re-validate datasets as legal and regulatory conditions change.
• Report key risk indicators (unapproved components found, unpatched critical CVEs, models without model cards) to governance bodies.

5. Frameworks and References to Recognise
• OWASP Top 10 for LLM Applications – Supply Chain Vulnerabilities; Data and Model Poisoning; Excessive Agency (plugins/tools).
• MITRE ATLAS – adversary techniques such as poisoning training data, publishing poisoned models and supply-chain compromise.
• NIST AI RMF – Govern / Map / Measure / Manage; emphasis on documentation, provenance and third-party considerations.
• NIST SSDF and SLSA – secure software supply chain and build provenance levels.
• ISO/IEC 42001 – AI management system requirements including supplier and data management controls; ISO/IEC 23894 – AI risk management guidance.
• SBOM / CycloneDX ML-BOM – machine-readable inventory formats.
• EU AI Act – documentation and transparency obligations, copyright-compliance expectations for general-purpose AI providers.

6. How to Answer Exam Questions on This Topic

ISACA questions are scenario based and ask for the BEST, MOST important, PRIMARY or FIRST action. Apply this reasoning pattern:

Identify the risk category – Is the scenario about a model (backdoor, unsafe file, licence), a dataset (poisoning, PII, copyright) or a component (CVE, malicious package, plugin)? Then match the control that addresses the root of that risk.

Think governance before technology – When options mix a policy/inventory/ownership answer with a purely technical one, ISACA usually favours the governance answer as MOST important or FIRST, because controls must flow from policy and knowing what you have.

Sequence correctly – FIRST usually means identify / inventory / assess, not remediate. You cannot scan or patch what you have not inventoried; you cannot select controls before assessing risk.

Accountability never transfers – Any option implying that the open-source community, the hub, or the original author is responsible for the organisation's risk is a distractor.

Risk-based and proportionate – Answers that apply the same heavy control to everything, or ban all open source outright, are rarely BEST. Prefer answers tied to criticality, risk appetite and documented acceptance.

Evidence over assertion – Published benchmarks, download counts and popularity are not assurance. Independent testing, signatures, hashes and documentation are.

Management perspective – You are answering as a security manager, not an ML engineer. Choose the option that establishes process, ownership, documentation and oversight over the one that names a specific tool.

7. Exam Tips: Answering Questions on Open-Source Models, Datasets and Component Risk

• Tip 1 – Inventory first. If asked what should be done FIRST to manage open-source AI risk, the answer is almost always to establish an inventory / AI-BOM of models, datasets and dependencies.
• Tip 2 – Provenance beats popularity. The BEST way to gain assurance over a downloaded model is to verify source, integrity (hash/signature) and documentation, then test it independently – not to rely on community ratings.
• Tip 3 – Unsafe serialisation is a code-execution risk. If a scenario mentions loading model files from untrusted sources, think malware scanning, safe formats and sandboxed loading.
• Tip 4 – Backdoors evade normal testing. If a model passes accuracy benchmarks but came from an unverified source, the residual risk is still high; the correct answer involves provenance controls and targeted adversarial/backdoor testing, not accepting the benchmark.
• Tip 5 – Licence is a risk, not just a legal detail. Open-weight models with non-commercial or restricted-use licences create compliance risk; the BEST control is legal review before adoption and recording obligations in the inventory.
• Tip 6 – Dataset questions hinge on lineage. For poisoning, PII or copyright scenarios, favour answers about data provenance, validation pipelines, consent basis and documentation (datasheets) over answers that only mention more training.
• Tip 7 – Component risk equals software supply-chain risk plus AI specifics. Pinning versions, internal mirrors, SBOM, vulnerability scanning and signed builds are the expected controls; dependency confusion and typosquatting are classic distractor-revealing terms.
• Tip 8 – Integrate, do not duplicate. When asked how to govern open-source AI, prefer extending existing TPRM, SDLC and change-management processes to cover models and data rather than building isolated AI-only processes.
• Tip 9 – Residual risk needs an owner. If a scenario ends with risk that cannot be fully mitigated, the correct answer involves documented acceptance by the appropriate business/risk owner, not the security team silently accepting it.
• Tip 10 – Monitoring is continuous. A one-time assessment at adoption is insufficient; answers that include ongoing vulnerability monitoring, re-scanning on update and behavioural monitoring in production are stronger.
• Tip 11 – Eliminate absolutes. Options such as ban all open-source models or trust all models from a reputable hub are almost always wrong.
• Tip 12 – Watch the qualifier. MOST important often points to policy/risk-appetite alignment; BEST control points to the most direct mitigation of the stated risk; FIRST points to identification or assessment steps.

8. Worked Example Questions

Example 1. A development team has downloaded a popular open-weight LLM from a public hub for a customer-facing application. Which of the following should the AI security manager do FIRST?
A. Deploy output filtering to block harmful responses.
B. Verify the model's provenance, integrity and licence terms.
C. Fine-tune the model on internal data.
D. Conduct a penetration test of the application.
Answer: B. Provenance, integrity and licence verification is the foundational due-diligence step; filtering and testing are later-stage controls, and fine-tuning does not address inherited risk.

Example 2. An organisation plans to fine-tune a model using a publicly available dataset. What is the GREATEST risk the security manager should address?
A. Increased training cost.
B. Dataset poisoning and unknown data provenance.
C. Longer model training time.
D. Reduced model accuracy on edge cases.
Answer: B. Poisoning and lack of provenance create security, legal and integrity exposure; the others are operational or performance concerns.

Example 3. Which of the following BEST enables an organisation to respond quickly when a vulnerability is disclosed in a widely used ML framework?
A. A documented AI acceptable-use policy.
B. Annual third-party audits of AI vendors.
C. A maintained AI-BOM listing components and versions for each AI system.
D. Mandatory security awareness training for data scientists.
Answer: C. Rapid response depends on knowing exactly which systems contain the affected component and version.

Example 4. A security scan shows that a model file obtained from a community repository uses a serialisation format capable of executing arbitrary code. What is the BEST course of action?
A. Load the model on a production server to verify it works.
B. Convert or obtain the model in a safe format and scan it, loading only in an isolated environment.
C. Rely on the repository's download count as evidence of safety.
D. Accept the risk because the model is widely used.
Answer: B. Treat the file as untrusted code: safe format, scanning and sandboxing directly mitigate the execution risk.

9. Key Takeaways
• Open-source models, datasets and components form an AI supply chain; the organisation inherits their risks but keeps full accountability.
• Core risk categories: backdoored or unsafe models, poisoned or non-compliant datasets, vulnerable or malicious components, and licence/provenance gaps.
• Core management actions: govern (policy, ownership, risk appetite) → inventory (AI-BOM) → assess (provenance, documentation, licence, scanning, independent testing) → treat (avoid/mitigate/transfer/accept) → monitor (continuous).
• In the exam, favour governance-first, inventory-first, evidence-based, risk-proportionate answers, and reject options that transfer accountability, rely on popularity, or apply absolutes.