Foundation Models and Large Language Models

1. Why This Topic Matters

Foundation models and large language models (LLMs) are the single biggest reason that AI risk has moved from a niche technical concern to a board-level governance issue. Until roughly 2020, most organizations deployed narrow AI systems built for a single purpose, trained on data they owned, and tested against a defined set of outcomes. Foundation models broke that pattern. One model, trained by a third party on vast internet-scale data, can now be adapted to hundreds of downstream uses, most of which the model developer never anticipated. This shifts where risk lives, who owns it, and how it can be controlled.

For the ISACA Advanced in AI Risk (AAIR) credential, this topic sits inside the domain AI Risk Governance and Framework Integration because foundation models stress-test every traditional governance assumption: data provenance, model explainability, third-party risk, accountability across the supply chain, and the ability to map controls to frameworks such as the NIST AI Risk Management Framework (AI RMF), ISO/IEC 42001, ISO/IEC 23894, the EU AI Act, and ISACA's own Digital Trust Ecosystem Framework (DTEF). An AI risk professional who cannot articulate how a general-purpose model changes the risk profile of an organization cannot credibly govern it.

Expect the exam to test this topic from a governance and risk-practitioner perspective, not a data-science perspective. You will not be asked to compute attention weights. You will be asked to decide which risk is most relevant, which control is most appropriate, which party is accountable, and how a framework should be applied.

2. What Foundation Models and LLMs Are

Foundation model: A large-scale model trained on broad data (usually through self-supervised learning) that can be adapted, through fine-tuning, prompting, or other methods, to a wide range of downstream tasks. The term was popularized by the Stanford Center for Research on Foundation Models in 2021. The defining properties are scale, generality, and adaptability. Foundation models can be text-based, image-based, audio-based, or multimodal (handling several data types at once).

Large language model (LLM): A foundation model specialized for natural language. It predicts the next token (word fragment) given preceding context and, at sufficient scale, exhibits capabilities such as summarization, translation, reasoning-like behavior, code generation, and conversational interaction. Examples include the GPT, Claude, Gemini, Llama, and Mistral families. Every LLM is a foundation model, but not every foundation model is an LLM.

Generative AI: An umbrella term for AI systems that produce new content (text, images, code, audio, video). Most modern generative AI is built on foundation models. The exam may use these terms close together; know that generative AI describes what the system does, foundation model describes how it was built and its general-purpose nature, and LLM describes a language-focused subtype.

General-purpose AI (GPAI): The term used in the EU AI Act for models that display significant generality and can perform a wide range of tasks. The Act imposes specific obligations on GPAI model providers (technical documentation, copyright compliance, training data summaries) and heavier obligations on GPAI models with systemic risk, presumed when training compute exceeds a defined threshold (10^25 floating-point operations). If a question mentions GPAI, it is pointing at foundation models under EU regulation.

3. How Foundation Models Work (Governance-Relevant Mechanics)

You need enough technical understanding to reason about risk. The key stages are:

a) Pre-training. The model ingests enormous datasets (web crawls, books, code repositories, licensed corpora) and learns statistical patterns through self-supervised objectives such as next-token prediction. Risk implications: training data provenance is often opaque; copyrighted, personal, biased, or toxic content may be embedded; the organization deploying the model usually did not control this stage.

b) Architecture. Nearly all current LLMs use the transformer architecture, which relies on an attention mechanism to weigh relationships between tokens. Parameters (weights) number in the billions to trillions. Risk implications: the model is a black box; explainability is limited to post-hoc techniques; behavior is probabilistic rather than deterministic, so identical inputs can yield different outputs.

c) Alignment and fine-tuning. After pre-training, developers apply supervised fine-tuning, reinforcement learning from human feedback (RLHF), constitutional or rule-based methods, and safety training to make outputs more helpful and less harmful. Organizations may also fine-tune on domain data. Risk implications: alignment is imperfect and can be circumvented (jailbreaks); fine-tuning on proprietary data creates new data protection and leakage exposures; fine-tuning can degrade safety behaviors that were present in the base model.

d) Adaptation and deployment patterns. Organizations consume foundation models through several patterns, each with a distinct risk profile:
- Direct API access to a vendor-hosted model (third-party and data-transfer risk dominate).
- Prompt engineering and system prompts to steer behavior (prompt injection and instruction leakage risk).
- Retrieval-Augmented Generation (RAG), where the model is given relevant documents at query time to ground answers (access control, data poisoning, and permission-bypass risk).
- Fine-tuning on proprietary data (model drift, memorization, IP exposure).
- Open-weight self-hosting (greater control but full responsibility for security, patching, and misuse).
- Agentic deployment, where the model is given tools, memory, and the ability to take actions (autonomy, cascading failure, and accountability risk).

e) Inference and output. Users or systems send prompts; the model generates responses token by token. Risk implications: hallucination (confident, fluent, false output), inconsistent quality, sensitive data disclosure, and inappropriate content.

4. The Foundation Model Risk Landscape

Exam questions will frequently ask you to identify the most significant or most relevant risk. Learn these categories and their distinguishing features:

Hallucination and factual unreliability. The model generates plausible but incorrect content. Most dangerous in high-stakes domains (legal, medical, financial) and when outputs are consumed without human review. Control: human-in-the-loop review, RAG grounding, output verification, citation requirements, restricting use cases.

Prompt injection. Malicious instructions embedded in user input or in retrieved content (indirect injection) override the system's intended behavior. This is the leading entry in the OWASP Top 10 for LLM Applications. Control: input and output filtering, privilege separation between model and tools, treating all retrieved content as untrusted, least-privilege tool permissions.

Sensitive information disclosure and data leakage. Training data memorization, employees pasting confidential data into public tools, and RAG systems surfacing documents users should not see. Control: data classification policies, acceptable use policies, enterprise agreements prohibiting training on customer data, access-controlled retrieval, data loss prevention tooling.

Bias and discriminatory output. Inherited from training data and amplified at scale. Control: bias testing across protected groups, red teaming, diverse evaluation sets, impact assessments.

Intellectual property and copyright. Both inbound (training on copyrighted works) and outbound (generating infringing content). Control: vendor indemnification clauses, licensing review, output screening, legal review of use cases.

Supply chain and third-party risk. The organization depends on a model it did not build, cannot inspect, and that may change without notice (silent model updates). Control: vendor due diligence, model cards and system cards, contractual change-notification rights, version pinning, independent evaluation, exit strategies.

Model poisoning and backdoors. Adversaries corrupt training or fine-tuning data, or distribute tampered open-weight models. Control: provenance verification, trusted repositories, integrity checks, evaluation before deployment.

Over-reliance and automation bias. Users trust fluent output uncritically. Control: training and awareness, user interface design that signals uncertainty, mandatory review for consequential decisions.

Misuse and dual use. Models assisting in fraud, disinformation, malware creation, or harassment. Control: acceptable use policies, usage monitoring, abuse detection, alignment with vendor usage policies.

Emergent and unpredictable capabilities. Capabilities appear at scale that were not explicitly designed, complicating risk assessment. Control: ongoing evaluation, red teaming, staged rollout, capability thresholds triggering additional governance.

Environmental and concentration risk. Large compute and energy footprints; dependence on a small number of providers. Control: sustainability reporting, multi-vendor strategies, consideration in ESG governance.

5. Governance and Framework Integration

This is the core of the AAIR domain. You must be able to connect foundation model risk to formal governance structures.

Shared responsibility across the AI value chain. Foundation models create a layered ecosystem: the model developer (trains the base model), the deployer or provider of a downstream system (builds a product on it), and the end user. Accountability must be explicitly allocated. A recurring exam theme: the deploying organization cannot outsource accountability to the model vendor. It remains responsible for the outcomes of the system it puts in front of its customers and employees, even if it can contractually transfer some liability.

NIST AI RMF. The four functions, Govern, Map, Measure, Manage, apply directly. For foundation models: Govern establishes policies and roles for generative AI; Map documents the intended use, the base model's known limitations, and the downstream context; Measure evaluates hallucination rates, bias, security, and performance; Manage prioritizes and treats risks and monitors post-deployment. NIST also published a Generative AI Profile (NIST AI 600-1) that enumerates twelve risk categories unique to or exacerbated by generative AI, including confabulation, data privacy, information integrity, information security, harmful bias, intellectual property, and value chain and component integration. If a question references a NIST generative AI profile, this is the document.

ISO/IEC 42001 (AI Management System). Provides a Plan-Do-Check-Act management system for AI. For foundation models, the relevant controls include AI policy, roles and responsibilities, impact assessment, data management for AI, supplier relationships, and lifecycle management. The standard explicitly contemplates organizations that use third-party AI components, which is exactly the foundation model scenario.

ISO/IEC 23894 (AI Risk Management). Extends ISO 31000 to AI. Emphasizes identifying AI-specific risk sources such as lack of transparency, complexity of the environment, and automation. Useful for structuring a foundation model risk register.

EU AI Act. Introduces the GPAI model tier. Providers of GPAI models have transparency, documentation, and copyright obligations; providers of GPAI models with systemic risk must additionally perform model evaluations, adversarial testing, incident reporting, and cybersecurity protections. Downstream deployers of high-risk systems built on GPAI retain their own obligations. Remember the distinction between provider and deployer and the fact that generative AI outputs such as deepfakes carry transparency labeling requirements.

ISACA guidance and DTEF. ISACA frames AI through digital trust: the organization must be able to demonstrate that AI use is trustworthy, accountable, secure, and aligned to stakeholder expectations. COBIT-style governance principles still apply: define objectives, assign ownership, measure performance, and ensure oversight by the board.

Practical governance mechanisms for foundation models:
- A generative AI acceptable use policy specifying approved tools, prohibited data, and required review.
- An AI inventory or registry capturing every foundation model use, including shadow AI uncovered through discovery.
- Use-case risk tiering so that low-risk internal drafting is treated differently from customer-facing or decision-making uses.
- AI impact assessments before deployment, revisited on material change.
- Vendor due diligence covering training data practices, data retention, security certifications, model documentation, and change notification.
- Red teaming and adversarial testing before and after release.
- Monitoring for drift, misuse, hallucination rates, and incidents, with defined escalation paths.
- Human oversight proportionate to the consequence of the output.
- Clear communication to users that they are interacting with AI and about its limitations.

6. How to Approach Exam Questions on This Topic

AAIR questions are scenario-based and typically ask for the best, most, first, or primary answer. Use the following reasoning process:

Step 1: Identify the deployment pattern. Is the organization using a vendor API, self-hosting open weights, fine-tuning, building a RAG application, or deploying agents? The pattern determines which risks dominate. Vendor API scenarios point to third-party and data-transfer controls. RAG scenarios point to access control and indirect prompt injection. Agentic scenarios point to autonomy limits and least privilege.

Step 2: Identify the stakeholder perspective. Are you answering as the risk practitioner, the board, the auditor, or the model provider? The AAIR exam almost always wants the risk practitioner view: assess, advise, govern, and escalate rather than build or code.

Step 3: Locate the question within the lifecycle. Pre-deployment questions favor impact assessment, due diligence, and testing. Post-deployment questions favor monitoring, incident response, and periodic review.

Step 4: Apply governance-first logic. When choices include a technical fix, a policy or governance action, and a do-nothing option, the exam generally rewards the answer that establishes accountability, defines scope, or performs an assessment before implementing point solutions. Technical controls are right when the question clearly asks for a mitigation of a specific identified threat.

Step 5: Eliminate answers that violate core principles. Wrong answers often: transfer accountability entirely to a vendor; assume the model is deterministic or fully explainable; ban AI outright when risk-proportionate controls are available; rely solely on user training when systemic controls exist; or treat generative AI identically to traditional software.

7. Exam Tips: Answering Questions on Foundation Models and Large Language Models

Tip 1: Know the vocabulary precisely. Foundation model, LLM, generative AI, GPAI, fine-tuning, RAG, prompt injection, hallucination (also called confabulation in NIST terminology), RLHF, model card, system card, open-weight, agentic AI. Distractors exploit near-synonyms.

Tip 2: Accountability never leaves the deployer. If an answer option suggests that using a reputable vendor removes the organization's responsibility for outcomes, it is wrong. Contracts allocate liability; governance allocates accountability.

Tip 3: Hallucination is a reliability risk, not a security risk. Questions about false or fabricated output are about accuracy, validation, and human review. Questions about manipulated instructions, data exfiltration, or jailbreaks are security risks. Match the control category to the risk category.

Tip 4: Prompt injection is the signature LLM security threat. When content from an untrusted source (email, web page, document) is fed to a model that has tool access, the best answer usually involves treating that content as untrusted, restricting tool privileges, and separating instructions from data.

Tip 5: Public tools plus confidential data equals data leakage. Scenarios about employees pasting customer records or source code into consumer chatbots are testing acceptable use policy, data classification, enterprise licensing with no-training clauses, and data loss prevention.

Tip 6: Prefer risk-proportionate tiering over blanket bans. The exam favors governance that enables responsible use. Answers that recommend classifying use cases by risk and applying graduated controls usually beat answers that prohibit all generative AI.

Tip 7: Model updates are a change management event. Vendors update hosted models frequently. A model that behaves differently after an unannounced update is a third-party change management and monitoring issue. Controls: contractual notification, version pinning, regression testing, continuous evaluation.

Tip 8: Explainability expectations must be realistic. Foundation models cannot be fully explained. Questions asking how to satisfy transparency obligations should be answered with documentation (model and system cards), disclosure to users, logging, testing evidence, and human oversight, not with promises of complete interpretability.

Tip 9: Map to the framework the question names. If the question says NIST AI RMF, answer with Govern, Map, Measure, Manage. If it says ISO/IEC 42001, think management system, policy, impact assessment, continual improvement. If it says EU AI Act, think GPAI provider obligations, systemic risk, provider versus deployer, transparency labeling. Do not mix framework language unless the question asks about integration.

Tip 10: Agentic AI raises the stakes. When a model can take actions, the best answers emphasize least-privilege tool access, human approval for consequential actions, kill switches, action logging, and bounded scope.

Tip 11: Red teaming and evaluation are pre-deployment and ongoing. A question asking how to gain assurance over a foundation model's behavior before launch points to adversarial testing and evaluation against the intended use. A question about assurance after launch points to monitoring and periodic reassessment.

Tip 12: Read for the superlative. Underline most, best, first, primary, greatest. Several options may be true; only one best fits the qualifier. First usually means assess or define scope. Primary usually means the root cause or governing principle. Best control means the one most directly addressing the stated risk with the least collateral disruption.

Tip 13: Shadow AI is an inventory problem. If employees are using unapproved generative AI tools, the first governance step is discovery and inventory, followed by policy and approved alternatives, not immediate punitive measures.

Tip 14: Do not over-engineer the answer. The exam tests judgment, not technical depth. If an option requires the risk practitioner to retrain the model or modify its architecture, it is almost certainly a distractor.

8. Quick Reference Summary

- Foundation models are large, general-purpose, adaptable models; LLMs are the language subtype; generative AI describes the output behavior; GPAI is the EU regulatory label.
- Risk shifts to the value chain: developer, deployer, user. The deployer remains accountable.
- Top risks: hallucination, prompt injection, data leakage, bias, IP, third-party dependence, poisoning, over-reliance, misuse, emergent behavior.
- Top controls: acceptable use policy, AI inventory, use-case tiering, impact assessment, vendor due diligence, RAG with access control, human oversight, red teaming, monitoring, change management.
- Frameworks: NIST AI RMF and its Generative AI Profile, ISO/IEC 42001, ISO/IEC 23894, EU AI Act GPAI provisions, ISACA DTEF and COBIT principles.
- Exam mindset: governance first, accountability stays in-house, proportionate controls over bans, match control type to risk type, honor the superlative in the question.