Fine-Tuning, RAG and Prompt Design Risks

1. Why This Topic Matters

Most organisations do not build foundation models from scratch. Instead, they adapt a pre-trained model to their business needs using one or more of three techniques: fine-tuning, Retrieval-Augmented Generation (RAG) and prompt design (prompt engineering). Each technique introduces a distinct layer of risk that sits on top of the risks already inherent in the base model. ISACA's Advanced in AI Risk (AAIR) framework places this content inside the AI Life Cycle Risk Management domain because these adaptation choices are made during the design, development and deployment phases and they directly determine the organisation's exposure to data leakage, manipulation, inaccurate outputs, intellectual property problems and regulatory non-compliance.

For a risk professional, the critical insight is that the adaptation method is a control decision. Choosing fine-tuning versus RAG versus prompting changes where sensitive data lives, who can influence model behaviour, how auditable the system is and how quickly errors can be corrected. Exam questions test whether you can identify the risk that is specific to each method and select the control that addresses it at the right stage of the life cycle.

2. What It Is – The Three Adaptation Approaches

Fine-tuning means continuing the training of a pre-trained model on a smaller, task-specific dataset so that the model's internal weights change. Variants include full fine-tuning, parameter-efficient fine-tuning (PEFT such as LoRA or adapters) and instruction or preference tuning (RLHF, DPO). Knowledge becomes baked into the model.

Retrieval-Augmented Generation (RAG) keeps the model weights unchanged. At inference time the system retrieves relevant documents from an external knowledge store (often a vector database built from embeddings) and injects them into the prompt as context so the model can ground its answer in current, organisation-specific information. Knowledge lives outside the model and can be updated without retraining.

Prompt design / prompt engineering shapes behaviour purely through the instructions supplied to the model: system prompts, few-shot examples, output format constraints, guardrail instructions and prompt templates. Nothing about the model or the data store changes; behaviour is steered at the interface.

A simple way to remember the hierarchy: Prompting is cheapest and least durable; RAG adds knowledge without changing the model; fine-tuning changes the model itself and is the most expensive and least reversible.

3. How It Works – The Risk Profile of Each Method

3.1 Fine-Tuning Risks

Training data poisoning. Because fine-tuning data directly alters weights, malicious or low-quality records (including hidden backdoor triggers) can permanently embed unwanted behaviour. Risk is highest when datasets are crowdsourced, scraped or supplied by third parties.

Memorisation and privacy leakage. Fine-tuned models can regurgitate verbatim training records (PII, trade secrets, customer data). This creates GDPR, HIPAA and confidentiality exposure and makes data subject deletion requests extremely difficult because the data cannot simply be removed from the weights.

Catastrophic forgetting and capability regression. Narrow fine-tuning can degrade general capabilities or erase safety alignment that the base model vendor established. Research shows even benign fine-tuning can weaken refusal behaviours, so safety evaluations must be re-run after every fine-tuning cycle.

Overfitting and bias amplification. Small, unrepresentative datasets amplify existing bias and reduce generalisation. Fairness testing performed on the base model is not transferable to the fine-tuned version.

Licensing and IP risk. Base model licences may restrict fine-tuning, redistribution or commercial use; training data may carry copyright or contractual restrictions. Derivative-work ownership can be unclear.

Version and lineage risk. Each fine-tuned checkpoint is a new model artefact requiring its own model card, lineage record, approval and rollback plan. Without a model registry, organisations lose track of which version is in production and what data produced it.

Cost and reversibility. Correcting a fine-tuning mistake requires retraining; there is no quick patch. This makes fine-tuning the least agile method for rapidly changing or highly regulated knowledge.

3.2 RAG Risks

Knowledge-base poisoning and indirect prompt injection. Because retrieved documents are placed into the prompt, any attacker who can insert or modify a document in the knowledge store (a shared drive, wiki, web page, email archive) can plant instructions that the model may follow. This is the signature RAG threat and appears in the OWASP Top 10 for LLM Applications.

Access-control bypass and data leakage. If the retrieval layer does not enforce the same permissions as the source systems, a user can obtain information they are not entitled to see simply by asking. Document-level and row-level authorisation must be propagated into the vector store and enforced at query time.

Stale, inaccurate or unauthorised sources. RAG is only as trustworthy as its corpus. Outdated policies, draft documents, duplicates and uncurated content lead to confidently wrong answers that appear grounded.

Hallucination despite grounding. Retrieval reduces but does not eliminate fabrication. The model may ignore retrieved context, blend it with parametric memory or cite sources that do not support the claim. Faithfulness and citation accuracy need to be measured, not assumed.

Embedding and vector database security. Embeddings can be inverted to recover source text; vector databases are often new, less-hardened infrastructure with weak default authentication, logging and encryption.

Retrieval quality failures. Poor chunking, weak embedding models or irrelevant top-k results produce degraded answers. This is an operational risk that requires retrieval metrics (precision, recall, relevance) in monitoring.

Provenance and auditability. A strength of RAG is that sources can be cited; a risk is that organisations fail to log which documents were retrieved for each answer, leaving no evidence trail for audits, disputes or incident investigation.

3.3 Prompt Design Risks

Direct prompt injection and jailbreaking. Users craft inputs that override system instructions, extract restricted content or force the model to act outside policy. Prompts are natural language, so instruction and data are not cleanly separated, which is the root cause.

System prompt leakage. Confidential instructions, internal rules, hidden tool definitions or embedded credentials can be extracted by asking the model to reveal its prompt. Secrets must never be stored in prompts.

Over-reliance on prompts as a security control. Instructions such as 'never reveal customer data' are probabilistic guidance, not enforcement. Treating a prompt as the sole guardrail is a control design weakness; deterministic controls (input filtering, output filtering, authorisation, rate limits) must sit outside the model.

Brittleness and silent drift. Small wording changes, model version upgrades by the vendor or context-window pressure can change behaviour unpredictably. Prompts need version control, regression testing and change management like any other code.

Sensitive data in prompts. Users and applications routinely paste confidential or personal data into prompts. If prompts are logged, sent to third-party APIs or used by the vendor for training, this creates confidentiality and cross-border transfer exposure.

Manipulative or biased framing. Prompt wording can steer outputs toward biased, misleading or non-compliant content, intentionally or not. Prompt templates deserve the same review as policy text.

Excessive agency. When prompts grant the model access to tools, plug-ins or APIs, a successful injection can turn into unauthorised actions (sending emails, executing transactions). Least privilege and human-in-the-loop approval for consequential actions are essential.

4. Mapping Risks and Controls Across the AI Life Cycle

Design / Requirements phase: Choose the adaptation method deliberately based on data sensitivity, update frequency, explainability needs and regulatory obligations. Document the rationale in the risk register. Perform a threat model that includes injection, poisoning and leakage paths specific to the chosen method.

Data phase: For fine-tuning – data provenance, licensing review, de-identification, quality and bias checks, poisoning screening. For RAG – corpus curation, classification labelling, ownership, retention and permission mapping before ingestion.

Development / Build phase: Prompt and template version control; separation of system instructions from user content; parameterised prompts; secrets kept in vaults not prompts; safety and alignment re-evaluation after fine-tuning; red-team testing for injection and jailbreaks; retrieval relevance and faithfulness testing.

Validation / Pre-deployment phase: Independent evaluation against acceptance criteria; bias and fairness testing on the adapted model; privacy testing for memorisation (membership inference, extraction attempts); penetration testing of the vector store and orchestration layer; documentation updates (model card, data sheet, system card).

Deployment / Operations phase: Input and output guardrails outside the model; authorisation enforced at retrieval time; logging of prompts, retrieved sources and outputs with appropriate privacy protections; monitoring for drift, hallucination rates and anomalous prompt patterns; incident response playbooks for poisoning, injection and leakage; periodic re-indexing and corpus hygiene; change management for prompt and model updates.

Decommissioning phase: Secure deletion of fine-tuning datasets, checkpoints, embeddings and prompt logs in line with retention policy; confirm data subject rights obligations can be met.

5. Quick Comparison for Recall

Where knowledge lives: Fine-tuning – in model weights; RAG – in external documents and vectors; Prompting – in the instruction text.
Ease of update: Fine-tuning – low (retrain); RAG – high (update corpus); Prompting – very high (edit text).
Signature risk: Fine-tuning – memorisation, poisoning baked in, lost alignment; RAG – indirect injection via documents and access-control bypass; Prompting – direct injection, jailbreak, system prompt leakage, false sense of control.
Explainability / traceability: Fine-tuning – low; RAG – higher through citations and retrieval logs; Prompting – moderate (behaviour depends on wording and model version).
Primary mitigation theme: Fine-tuning – data governance and post-tuning safety evaluation; RAG – corpus governance, permission-aware retrieval, sanitisation of retrieved content; Prompting – defence in depth with controls outside the model, version control and testing.

6. Exam Tips: Answering Questions on Fine-Tuning, RAG and Prompt Design Risks

Tip 1 – Identify the method first. Read the scenario and determine whether the organisation changed the weights (fine-tuning), added an external knowledge store (RAG) or only wrote instructions (prompting). The correct answer is almost always the risk or control that is unique to that method.

Tip 2 – Match the keyword to the risk. 'Verbatim reproduction of training records' or 'cannot honour deletion request' points to fine-tuning memorisation. 'Instructions hidden in a document the model retrieved' is indirect prompt injection (RAG). 'User asked the chatbot to ignore previous instructions' is direct prompt injection. 'Model reveals its hidden instructions' is system prompt leakage. 'Employee saw a document they are not authorised to access' is RAG access-control failure.

Tip 3 – Remember that prompts are not security controls. If an answer option says the organisation should rely on instructing the model not to do something, it is usually the wrong (or weakest) answer. Prefer options that place deterministic controls outside the model: input/output filtering, authorisation, least-privilege tool access, human approval, logging.

Tip 4 – Think life cycle stage. ISACA frames risk by when it should be addressed. Questions asking for the most effective or best control usually favour earlier, preventive measures (data governance, threat modelling, permission mapping before ingestion) over later detective measures (monitoring after deployment). But if the scenario is already in production, select operational controls (monitoring, re-indexing, incident response).

Tip 5 – Safety does not carry over. A frequent trap is assuming the vendor's alignment, bias testing or safety assurances still apply after fine-tuning. The correct answer is to re-evaluate the adapted model.

Tip 6 – RAG is a data-governance problem as much as an AI problem. When a RAG scenario mentions stale, wrong or sensitive answers, look for answers about corpus curation, classification, ownership, retention and permission-aware retrieval rather than retraining the model.

Tip 7 – Reversibility and agility. If the question stresses frequently changing information or regulatory content that must be current and citable, RAG is preferred over fine-tuning. If it stresses style, format or domain language adaptation with stable knowledge, fine-tuning is reasonable, provided data governance is strong.

Tip 8 – Treat prompts and corpora as change-controlled artefacts. Answers that mention version control, regression testing, documentation and approval for prompt templates and knowledge bases align with ISACA's governance mindset and are usually correct over ad hoc approaches.

Tip 9 – Third-party and vendor angle. Look for risks around sending prompts and retrieved data to external APIs: vendor training on your data, data residency, licensing of the base model and lack of audit rights. Contractual controls and data-use restrictions are typical correct answers.

Tip 10 – Use the risk-management vocabulary. Frame your reasoning in terms of threat, vulnerability, impact, control objective and residual risk. The AAIR exam rewards answers that describe the most appropriate control given the risk, not the most technically sophisticated option. When two answers seem correct, pick the one that is preventive, independent of the model's own behaviour and documented.

7. One-Line Summary

Fine-tuning embeds risk permanently into the model, RAG moves risk into the knowledge store and retrieval path, and prompt design leaves risk at a fragile interface – so a mature AI risk programme selects the adaptation method deliberately, governs the data and instructions feeding it, tests the adapted system independently and enforces security with controls that live outside the model.