Security Architecture Patterns for AI Applications

1. Why Security Architecture Patterns for AI Applications Matter

AI applications introduce an attack surface that traditional application security was never designed to cover. A conventional web application processes structured, predictable inputs; an AI application ingests natural language, images, documents and third-party data, passes them through probabilistic models whose behaviour cannot be fully specified, and often grants those models the ability to call tools, query databases or trigger actions. This means threats such as prompt injection, training data poisoning, model theft, sensitive information disclosure through outputs, and excessive agency have no exact equivalent in legacy systems.

Within the ISACA AAISM (Advanced in AI Security Management) body of knowledge, the AI Technologies and Controls domain expects candidates to move beyond listing threats and to demonstrate how security is designed in through repeatable architectural patterns. Patterns matter because:

- They provide reusable, proven structures that organisations can apply consistently across many AI use cases rather than reinventing controls for every project.
- They translate abstract governance requirements (NIST AI RMF, ISO/IEC 42001, the EU AI Act, OWASP Top 10 for LLM Applications) into concrete technical placement of controls.
- They support defence in depth, acknowledging that no single control (for example a content filter) is reliable against adaptive adversaries.
- They enable auditors and security managers to assess an AI system by asking where each expected control lives in the architecture, which is exactly how AAISM frames many scenario questions.

2. What Security Architecture Patterns for AI Applications Are

A security architecture pattern is a documented, reusable solution to a recurring security problem, described in terms of components, their relationships, the trust boundaries between them, and the controls enforced at each boundary. For AI applications, patterns describe how to protect the full lifecycle: data collection and preparation, training and fine-tuning, model storage and registry, deployment and inference, retrieval and tool use, and monitoring and response.

The core idea is to treat the AI model as an untrusted component, or at least a semi-trusted one, and to wrap it with enforcement points that the model cannot bypass. The model is a probabilistic engine that can be manipulated; the architecture around it is deterministic and policy-driven. Security must therefore be enforced outside the model wherever possible.

3. The Key Patterns You Must Know

3.1 Layered Defence (Defence in Depth) for AI
Controls are placed at multiple layers: the data layer (provenance, cleansing, access control), the model layer (robust training, adversarial testing, model signing), the application layer (input validation, output filtering, business logic authorisation), the infrastructure layer (network segmentation, hardened containers, secrets management) and the governance layer (logging, monitoring, human oversight). The assumption is that any one layer may fail.

3.2 AI Gateway / Policy Enforcement Point Pattern
All traffic to and from models passes through a centralised gateway (sometimes called an LLM gateway, AI firewall or model proxy). The gateway performs authentication and authorisation of callers, rate limiting and quota control, prompt inspection for injection patterns and sensitive data, output inspection for data leakage, toxicity and policy violations, logging of every request and response, routing between approved models, and cost and usage attribution. This pattern provides a single choke point for policy, which is far more auditable than embedding controls in each application.

3.3 Input Validation and Prompt Hardening Pattern
Untrusted user input is separated from trusted system instructions. Techniques include structured prompt templates, delimiting user content, instruction hierarchy, canonicalisation of inputs, length and format checks, and classifier-based detection of jailbreak attempts. Critically, inputs arriving indirectly (web pages, emails, documents retrieved for RAG) are treated as untrusted data, never as instructions, to counter indirect prompt injection.

3.4 Output Filtering and Validation Pattern
Model outputs are inspected before reaching users or downstream systems. Controls include PII and secret detection, policy classifiers, schema validation of structured outputs (for example enforcing valid JSON before code consumes it), encoding of output to prevent cross-site scripting when rendered in a browser, and confidence thresholds or abstention when the model is uncertain. The principle is never trust model output as safe or correct.

3.5 Least Privilege and Tool Mediation Pattern (Agentic AI)
When a model can invoke tools, APIs or plugins, every tool call is mediated by a broker that enforces explicit allow-lists, scoped credentials, parameter validation and, for high-impact actions, human approval. The model never holds long-lived credentials; it requests an action and a deterministic component decides whether to execute it. This counters the OWASP risk of excessive agency.

3.6 Isolation and Sandboxing Pattern
Model inference, code execution generated by the model, and data processing pipelines run in isolated environments (containers, dedicated virtual networks, ephemeral sandboxes) with egress restrictions. Training environments are segregated from production inference. Multi-tenant deployments isolate tenants at the data, prompt and model-context level to prevent cross-tenant leakage.

3.7 Secure Retrieval-Augmented Generation (RAG) Pattern
In RAG architectures, security controls apply at the retrieval stage: documents are classified and tagged, the vector store enforces document-level access control so a user only retrieves content they are authorised to see, retrieved chunks are sanitised for embedded instructions, and provenance is attached to each chunk so answers can be traced to sources. Embedding inversion and vector store poisoning are additional threats addressed through access controls and integrity monitoring.

3.8 Zero Trust for AI Pipelines
No component of the pipeline is trusted by default. Every service-to-service call is authenticated with strong identity (mutual TLS, workload identity), authorised per request, encrypted in transit, and logged. This applies to data ingestion jobs, feature stores, training clusters, model registries, serving endpoints and monitoring tools.

3.9 Secure Model Supply Chain and MLOps Pattern
Models, datasets and dependencies are treated as software artefacts: signed, hashed, versioned and recorded in a model registry with an AI Bill of Materials (AI-BOM). Third-party and open-source models are scanned (for example for malicious serialised payloads), evaluated and approved before use. CI/CD pipelines enforce security gates such as adversarial robustness tests, bias evaluations and vulnerability scans before promotion to production.

3.10 Data Protection and Privacy Pattern
Training and inference data are protected through classification, minimisation, anonymisation or pseudonymisation, encryption at rest and in transit, and, where appropriate, privacy-enhancing technologies such as differential privacy, federated learning or confidential computing. Retention rules apply to prompts and logs, which often contain sensitive content.

3.11 Observability, Monitoring and Response Pattern
Prompts, responses, tool calls, retrieval results, latency, token usage and model performance metrics are logged to a central platform and monitored for anomalies such as spikes in refusals, unusual query patterns indicating model extraction, drift in output distributions, or repeated injection attempts. Alerts integrate with the SOC and incident response plans include AI-specific playbooks (for example model rollback or disabling a tool).

3.12 Human-in-the-Loop and Oversight Pattern
For high-risk decisions the architecture includes mandatory human review checkpoints, override capabilities, explainability outputs and clear escalation paths. This pattern directly supports regulatory requirements for human oversight and accountability.

4. How It Works: Following a Request Through a Secure AI Architecture

Consider a customer-facing assistant with RAG and tool access:

1. The user authenticates to the application; the session carries an identity and entitlements.
2. The request reaches the AI gateway, which validates the caller, applies rate limits and scans the prompt for injection, malware links and sensitive data.
3. The orchestrator constructs a hardened prompt: system instructions are kept separate from user text, and user text is clearly delimited.
4. The retrieval layer queries the vector store filtered by the user's entitlements, returning only authorised chunks, each with provenance metadata. Retrieved text is sanitised.
5. The model runs in an isolated inference environment with no direct network egress and no credentials.
6. If the model proposes a tool call, the tool broker checks the allow-list, validates parameters, applies scoped short-lived credentials and, for sensitive actions such as refunds, requests human approval.
7. The model output passes through output filters: PII redaction, policy classification, schema validation and encoding.
8. Every step is logged with correlation identifiers; monitoring detects anomalies and feeds the incident response process.
9. Separately, the MLOps pipeline ensures the model in production is signed, registered, tested and traceable to approved training data.

Notice that the model itself makes no security decisions. Every control that matters is enforced by a deterministic component at a trust boundary.

5. Mapping Patterns to Frameworks

- OWASP Top 10 for LLM Applications: prompt injection maps to input hardening and gateway; sensitive information disclosure maps to output filtering and RAG access control; supply chain vulnerabilities map to the secure MLOps pattern; excessive agency maps to tool mediation; insecure output handling maps to output validation and encoding.
- MITRE ATLAS: provides adversary tactics (reconnaissance, ML model access, poisoning, exfiltration) against which patterns are selected.
- NIST AI RMF (Govern, Map, Measure, Manage): patterns are the technical expression of the Manage function and are measured through monitoring.
- ISO/IEC 42001 and ISO/IEC 27001: patterns satisfy control objectives for AI system lifecycle, data management, supplier management and logging.
- EU AI Act: human oversight, logging, robustness and cybersecurity requirements for high-risk systems are met through the oversight, observability, isolation and supply chain patterns.

6. How to Answer Exam Questions on This Topic

AAISM questions are largely scenario-based and ask for the best, most effective, first or primary answer. Approach them as follows:

Step 1 - Identify the threat being described. Is the scenario about manipulated inputs (injection), leaked outputs (disclosure), unauthorised actions (excessive agency), compromised artefacts (supply chain), cross-tenant exposure (isolation) or undetected misuse (monitoring)?

Step 2 - Locate the trust boundary. Ask where the untrusted element enters and where the control should sit. The correct answer is almost always a control enforced outside the model at that boundary, not an instruction added inside the prompt.

Step 3 - Prefer architectural, preventive, centralised controls. When options include training the model to behave, writing a stronger system prompt, deploying a gateway with policy enforcement, or adding a user warning, the architectural control (gateway, broker, access-filtered retrieval) is normally the best answer. Prompt-only or awareness-only answers are typically distractors.

Step 4 - Apply the management lens. AAISM is a security management certification. If a question asks what the security manager should do first, the answer often involves risk assessment, defining requirements, establishing a reference architecture or ensuring governance, before selecting a specific tool.

Step 5 - Eliminate answers that trust the model. Any option assuming the model will reliably refuse, self-validate or keep secrets contradicts the core principle that models are probabilistic and manipulable.

7. Exam Tips: Answering Questions on Security Architecture Patterns for AI Applications

- Remember the golden rule: enforce security in deterministic components around the model, never rely on the model to enforce it.
- Separate instructions from data. Any retrieved or user-supplied content is data. If an answer treats retrieved documents as trusted instructions, it is wrong.
- Defence in depth beats single controls. If asked which approach is most effective against prompt injection, choose the combination of input filtering, output validation, least privilege and monitoring over any single measure.
- Least privilege is the answer to agentic risk. For scenarios where an AI agent performed an unauthorised or harmful action, the best control is tool mediation with scoped permissions and human approval for high-impact actions, not simply better prompts.
- Access control belongs at retrieval time. For RAG data leakage scenarios, the correct fix is document-level authorisation enforced in the retrieval layer before content reaches the model, not post-hoc output filtering alone.
- Supply chain questions point to signing, registries and AI-BOM. When a scenario involves a downloaded open-source model or dataset, think integrity verification, scanning for malicious serialisation, provenance records and an approval process.
- Centralisation aids auditability. A gateway pattern is frequently the best answer when the scenario mentions many teams, inconsistent controls, shadow AI, or difficulty demonstrating compliance.
- Logging must include prompts and outputs, but apply data protection to those logs. Questions on monitoring often test whether you recognise that logs themselves become sensitive data stores.
- Know the OWASP LLM Top 10 names and be ready to match each to its mitigating pattern, since question stems often paraphrase these risks without naming them.
- Distinguish preventive, detective and corrective controls. If the stem asks how to detect model extraction, the answer is monitoring of query patterns and rate limiting, not model signing.
- Watch for the words first, primary, best and most. First usually means assess or define requirements; best or most effective usually means the architectural control closest to the root cause.
- Do not over-engineer. If a scenario is a low-risk internal summarisation tool, answers imposing confidential computing and mandatory human review for every output are disproportionate; AAISM expects risk-based control selection.
- Link to governance. Reference architectures, standards and secure-by-design requirements are management tools; expect questions on how a security manager institutionalises patterns across the organisation, not only on how a single engineer implements them.

8. Quick Revision Summary

Security architecture patterns for AI applications are reusable designs that place deterministic controls at every trust boundary around a probabilistic model. The essential patterns are layered defence, the AI gateway, input hardening, output filtering, least-privilege tool mediation, isolation and sandboxing, secure RAG with access-filtered retrieval, zero trust pipelines, secure model supply chain, data protection, observability and human oversight. In the exam, identify the threat, find the boundary, choose the centralised preventive architectural control, apply a risk-based management perspective, and reject any option that trusts the model to secure itself.