AI System Components and Attack Surface
5 minutes
5 Questions
AI systems are composed of interconnected components, each of which expands the overall attack surface and requires dedicated security controls. The core components include: (1) Data pipelines, covering collection, labeling, storage, and preprocessing of training and inference data; (2) Models, including architecture, weights, hyperparameters, and fine-tuned derivatives; (3) Training infrastructure, such as compute clusters, orchestration frameworks, and experiment-tracking systems; (4) Inference and serving layers, including APIs, model gateways, and edge deployments; (5) Supporting components like vector databases, retrieval-augmented generation (RAG) stores, prompt templates, plugins, agents, and tool integrations; and (6) Monitoring, logging, and MLOps/LLMOps tooling. From an attack-surface perspective, each component introduces distinct threat vectors. Data pipelines are vulnerable to data poisoning, label manipulation, and supply-chain compromise of third-party datasets. Models face extraction (model theft via query access), inversion and membership inference (leaking training data), backdoor insertion, and adversarial examples that cause misclassification. Training infrastructure can be targeted through compromised dependencies, malicious pre-trained models from public hubs (e.g., pickle deserialization exploits), and insecure credentials. Inference endpoints are exposed to prompt injection (direct and indirect via retrieved content), jailbreaks, denial-of-wallet resource exhaustion, and output manipulation. Agentic systems with tool access amplify risk because a successful injection may trigger unauthorized actions, privilege escalation, or data exfiltration through connected systems. Vector stores and RAG pipelines can be poisoned to steer outputs or leak sensitive documents across tenants. Effective AI security management maps these components into a threat model (e.g., using MITRE ATLAS or the OWASP Top 10 for LLMs), applies defense-in-depth controls such as data provenance and integrity checks, model signing and SBOM/AIBOM tracking, input/output filtering and guardrails, least-privilege tool permissions, rate limiting, red teaming, and continuous monitoring for drift and anomalous queries. Understanding the full component inventory is essential because attackers exploit the weakest link, often in overlooked supporting infrastructure rather than the model itself.
AI System Components and Attack Surface: A Complete Guide for the ISACA AAISM Exam
Introduction
The topic AI System Components and Attack Surface sits inside the ISACA Advanced in AI Security Management (AAISM) Domain 3: AI Technologies and Controls. It is the foundation for almost everything else in that domain. You cannot select, justify, or audit a control unless you can first decompose an AI system into its parts and explain where each part can be attacked. On the exam, this concept appears both directly (identify the component or the threat) and indirectly (choose the best control for a scenario), so a solid mental model here pays off across many questions.
Why This Topic Is Important
1. AI systems are not just software. Traditional application security treats code as the primary asset. AI systems add data, learned parameters, probabilistic behavior, and often natural-language interfaces. Each of these introduces attack paths that classic security models (e.g., input validation only at the perimeter) do not cover.
2. The attack surface is larger and less visible. A model can be manipulated through its training data months before deployment, through a document it retrieves at run time, or through a third-party pretrained model downloaded from a public hub. Security managers must understand that the perimeter includes the entire data and model supply chain.
3. Governance frameworks require it. NIST AI RMF (Map and Measure functions), ISO/IEC 42001, the EU AI Act risk-based obligations, and ISACA guidance all expect organizations to inventory AI components and assess threats to each. The AAISM exam tests whether you can do this in a structured way.
4. Control selection depends on it. Choosing between data provenance checks, adversarial training, output filtering, rate limiting, or model access controls requires knowing which component is exposed and what the attacker is trying to achieve.
What It Is: The Components of an AI System
Think of an AI system as a pipeline plus the infrastructure that runs it. The AAISM exam generally expects you to recognize the following components:
1. Data layer
- Training data, validation data, and test data
- Fine-tuning datasets and human feedback data (RLHF)
- Retrieval sources (documents, knowledge bases, vector databases used in RAG)
- Inference inputs (prompts, images, sensor data, transaction records)
- Labels and annotations, often produced by third-party or crowd-sourced workers
2. Model layer
- Model architecture and hyperparameters
- Learned weights and parameters (the core intellectual property)
- Pretrained or foundation models obtained from vendors or open repositories
- Fine-tuned derivatives, adapters, and embeddings
- System prompts and guardrail configurations for LLMs
3. Development and training pipeline (MLOps)
- Feature engineering code, notebooks, and experiment tracking
- Training infrastructure (GPUs, cloud compute, orchestration)
- Model registries and artifact repositories
- CI/CD pipelines that package and promote models
- Evaluation and testing harnesses
4. Deployment and inference layer
- Model serving endpoints and APIs
- Inference infrastructure (containers, edge devices, serverless functions)
- Caching, load balancing, and rate limiting components
- Monitoring and logging for drift, performance, and misuse
5. Application and integration layer
- Front-end applications and chat interfaces
- Plugins, tools, function calling, and agent frameworks that let a model take actions
- Downstream systems that consume model outputs (databases, email, payment systems, code execution)
- Orchestration logic that chains multiple models or steps
6. Supply chain
- Open-source libraries and frameworks (PyTorch, TensorFlow, LangChain, etc.)
- Third-party models, datasets, and APIs
- Cloud AI services and managed platforms
- Model cards, licenses, and provenance documentation
7. Human and organizational layer
- Data scientists, ML engineers, and prompt engineers with privileged access
- Business users and end users who provide inputs and act on outputs
- Annotators and contractors
- Governance bodies that approve models for use
What It Is: The Attack Surface
The attack surface is the total set of points where an unauthorized actor can attempt to enter, manipulate, extract data from, or disrupt the system. In AI security, the attack surface is typically organized by lifecycle phase and by the asset being targeted. The exam frequently references two industry taxonomies: the OWASP Top 10 for LLM Applications and MITRE ATLAS (Adversarial Threat Landscape for AI Systems). You should be comfortable with the vocabulary from both.
Attacks on the data layer
- Data poisoning: injecting malicious or mislabeled samples into training or fine-tuning data to degrade performance or implant a backdoor (a trigger that produces attacker-chosen behavior).
- Label flipping: a poisoning variant that changes labels rather than inputs.
- Retrieval poisoning: planting malicious content in documents or web pages that a RAG system will retrieve, enabling indirect prompt injection.
- Data leakage in training sets: sensitive or regulated data (PII, PHI, secrets) included without authorization, later surfaced by the model.
Attacks on the model layer
- Evasion / adversarial examples: crafting inputs with small perturbations that cause misclassification at inference time (e.g., altered images, obfuscated malware, modified text).
- Model extraction / model stealing: querying an API repeatedly to reconstruct a functionally equivalent model or approximate its decision boundary.
- Model inversion: reconstructing representative training inputs (such as faces) from model outputs.
- Membership inference: determining whether a specific record was in the training set, which is a privacy violation.
- Model tampering / weight manipulation: directly altering stored weights or substituting a trojaned model in the registry.
- Backdoored pretrained models: downloading a foundation model that already contains hidden malicious behavior.
Attacks on LLM-specific components
- Direct prompt injection: the user supplies instructions that override the system prompt or safety policies (jailbreaks).
- Indirect prompt injection: instructions hidden in content the model processes (emails, web pages, PDFs, tool outputs) hijack the model without the user intending it.
- System prompt leakage: extracting confidential instructions, credentials, or business logic embedded in prompts.
- Insecure output handling: downstream systems trust model output and execute it (SQL, shell commands, HTML/JavaScript), leading to injection attacks.
- Excessive agency: an agent with too many tools, too much permission, or too much autonomy performs harmful actions when manipulated.
- Sensitive information disclosure: the model reveals memorized training data or data from other users (cross-tenant leakage).
- Unbounded consumption / denial of wallet: attackers trigger expensive queries to exhaust compute budgets or degrade availability.
Attacks on the pipeline and infrastructure
- Compromise of notebooks, CI/CD, or model registries to insert malicious code or artifacts
- Credential theft for cloud AI services
- Insecure deserialization of model files (e.g., pickle-based formats) leading to remote code execution
- Misconfigured storage buckets exposing datasets or weights
- Dependency and plugin vulnerabilities (software supply chain)
Attacks via the human layer
- Insider manipulation of training data or labels
- Social engineering of annotators or engineers
- Over-reliance by users who act on hallucinated or manipulated outputs (automation bias)
How It Works: Mapping Components to Threats and Controls
The practical skill the exam tests is threat modeling an AI system. A reliable method is a four-step approach:
Step 1: Inventory the components. Build an AI asset inventory that lists every model, dataset, pipeline, endpoint, plugin, and third-party dependency. Many organizations fail here because shadow AI (unsanctioned tools) is not captured. Governance frameworks consistently place inventory first.
Step 2: Identify trust boundaries and entry points. Where does untrusted content enter? User prompts, retrieved documents, third-party APIs, uploaded files, training data from external sources, and open-source models are all untrusted inputs. Where do outputs leave and what consumes them?
Step 3: Enumerate threats per component. Use MITRE ATLAS tactics (reconnaissance, resource development, initial access, ML attack staging, exfiltration, impact) or OWASP LLM categories as a checklist. Ask for each component: can it be poisoned, stolen, evaded, manipulated, leaked, or denied?
Step 4: Select layered controls. Map controls to the lifecycle:
- Data controls: provenance tracking, data validation and anomaly detection, access control on datasets, data minimization, differential privacy, sanitization of retrieval sources, content signing.
- Model controls: adversarial training and robustness testing, model watermarking and fingerprinting, encryption of weights at rest, integrity hashes and signed artifacts, secure model formats (e.g., safetensors over pickle), red teaming before release, model cards documenting intended use.
- Inference controls: authentication and authorization on endpoints, rate limiting and query monitoring to detect extraction attempts, input validation and prompt filtering, output filtering and content moderation, confidence thresholds with human review for high-risk decisions, logging of prompts and responses.
- Application and agent controls: least privilege for tools and plugins, human-in-the-loop approval for consequential actions, treating model output as untrusted input to downstream systems, sandboxing code execution, separating system instructions from user content.
- Pipeline controls: secure MLOps (code review, signed commits, protected registries), secrets management, segregation of duties between development and production, vulnerability management for dependencies, software bill of materials extended to an AI bill of materials (AIBOM).
- Supply chain controls: vendor due diligence, evaluation of third-party models and datasets, license review, contractual security requirements, monitoring of upstream advisories.
- Monitoring and response: drift detection, anomaly detection on query patterns, incident response playbooks for AI-specific events (poisoning discovered, jailbreak campaign, model leak), rollback capability to known-good model versions.
A Worked Example
Consider a customer-service chatbot built on a vendor LLM, augmented with RAG over internal policy documents, and able to issue refunds through an API.
- Components: vendor LLM (supply chain), system prompt (model layer), vector database of policy docs (data layer), orchestration code (application layer), refund API (downstream system), customer chat interface (entry point), support staff who upload documents (human layer).
- Attack surface: direct prompt injection by customers; indirect prompt injection via a maliciously edited policy document; excessive agency if the refund tool lacks limits; system prompt leakage revealing business rules; sensitive data disclosure if the vector store contains other customers' data; denial of wallet through long adversarial conversations.
- Priority controls: restrict who can modify the document store and validate uploads; cap refund amounts and require human approval above a threshold; apply output filtering and tool-call validation; isolate tenant data; rate limit sessions; log and monitor for jailbreak patterns; conduct red-team testing before launch.
The exam loves scenarios like this. The correct answer is almost always the control that addresses the specific exposed component at the earliest effective point, with a preference for preventive controls supported by detective monitoring.
Key Distinctions the Exam Expects You to Know
- Poisoning vs. evasion: poisoning occurs at training time and corrupts the model; evasion occurs at inference time and fools a correct model.
- Extraction vs. inversion vs. membership inference: extraction steals the model, inversion reconstructs training data, membership inference confirms whether a record was used.
- Direct vs. indirect prompt injection: direct comes from the user; indirect comes from content the model processes on the user's behalf.
- Insecure output handling vs. excessive agency: the first is about downstream systems trusting output; the second is about the model having too much power to act.
- Attack surface vs. vulnerability vs. threat: attack surface is the set of exposure points; a vulnerability is a specific weakness; a threat is the potential adverse event or actor.
Exam Tips: Answering Questions on AI System Components and Attack Surface
Tip 1: Decompose the scenario before reading the options. Identify which component is being discussed (data, model, pipeline, inference, application, supply chain, human). Most wrong answers are controls for the wrong component.
Tip 2: Pin the lifecycle phase. Ask whether the attack happens during data collection, training, deployment, inference, or operation. A question about corrupted labels is a training-phase issue; the answer will involve data validation or provenance, not output filtering.
Tip 3: Match vocabulary precisely. If the stem describes repeated API queries used to replicate a model, the term is model extraction, and the control is rate limiting plus query monitoring, not adversarial training. ISACA tests terminology carefully.
Tip 4: Treat all external content as untrusted. When a scenario involves documents, web pages, emails, or tool outputs feeding a model, think indirect prompt injection and look for controls on the content source and on tool permissions.
Tip 5: Prefer the earliest preventive control, backed by detection. If options include preventing a poisoned dataset from entering training versus detecting bad model behavior after deployment, prevention earlier in the lifecycle is usually the best answer, unless the question asks for a compensating or detective control.
Tip 6: Remember least privilege for agents. Questions about AI systems taking actions almost always resolve to limiting tool permissions, scoping credentials, and requiring human approval for high-impact actions.
Tip 7: Model output is input to something else. If a downstream system executes or renders model output, the risk is insecure output handling and the control is validation and sandboxing of that output.
Tip 8: Do not forget the supply chain. When a scenario mentions downloading a pretrained model or dataset from a public repository, think integrity verification, provenance, malicious serialization, and vendor due diligence.
Tip 9: Think like a security manager, not just an engineer. AAISM is a management certification. When two options are technically valid, choose the one that aligns with governance: inventory first, risk assessment, documented policies, accountable ownership, and monitoring. Governance and management answers often outrank purely technical fixes when the question asks what should be done first.
Tip 10: Watch for qualifiers. Words like first, most effective, primary, best, and greatest risk change the correct answer. First usually points to inventory, classification, or risk assessment. Most effective points to the control that eliminates the root exposure. Greatest risk points to the component with highest impact and least existing control.
Tip 11: Use frameworks as anchors. Recall OWASP Top 10 for LLM categories and MITRE ATLAS tactics to recognize the threat being described, and recall NIST AI RMF functions (Govern, Map, Measure, Manage) to place the appropriate management response.
Tip 12: Eliminate answers that solve traditional problems only. A firewall or standard web application scanner does not stop data poisoning or prompt injection. If an option is a classic IT control with no AI-specific effect on the described component, it is usually a distractor.
Quick Review Checklist
- Can I list the seven component groups of an AI system?
- Can I name at least two attacks against each component?
- Can I distinguish poisoning, evasion, extraction, inversion, and membership inference?
- Can I explain direct versus indirect prompt injection with an example?
- Can I map at least two controls to each lifecycle phase?
- Can I apply the four-step threat modeling approach to a new scenario in under two minutes?
If you can answer yes to each of these, you are well prepared for AAISM questions on AI System Components and Attack Surface, and you have built the foundation needed for the rest of Domain 3.
The topic AI System Components and Attack Surface sits inside the ISACA Advanced in AI Security Management (AAISM) Domain 3: AI Technologies and Controls. It is the foundation for almost everything else in that domain. You cannot select, justify, or audit a control unless you can first decompose an AI system into its parts and explain where each part can be attacked. On the exam, this concept appears both directly (identify the component or the threat) and indirectly (choose the best control for a scenario), so a solid mental model here pays off across many questions.
Why This Topic Is Important
1. AI systems are not just software. Traditional application security treats code as the primary asset. AI systems add data, learned parameters, probabilistic behavior, and often natural-language interfaces. Each of these introduces attack paths that classic security models (e.g., input validation only at the perimeter) do not cover.
2. The attack surface is larger and less visible. A model can be manipulated through its training data months before deployment, through a document it retrieves at run time, or through a third-party pretrained model downloaded from a public hub. Security managers must understand that the perimeter includes the entire data and model supply chain.
3. Governance frameworks require it. NIST AI RMF (Map and Measure functions), ISO/IEC 42001, the EU AI Act risk-based obligations, and ISACA guidance all expect organizations to inventory AI components and assess threats to each. The AAISM exam tests whether you can do this in a structured way.
4. Control selection depends on it. Choosing between data provenance checks, adversarial training, output filtering, rate limiting, or model access controls requires knowing which component is exposed and what the attacker is trying to achieve.
What It Is: The Components of an AI System
Think of an AI system as a pipeline plus the infrastructure that runs it. The AAISM exam generally expects you to recognize the following components:
1. Data layer
- Training data, validation data, and test data
- Fine-tuning datasets and human feedback data (RLHF)
- Retrieval sources (documents, knowledge bases, vector databases used in RAG)
- Inference inputs (prompts, images, sensor data, transaction records)
- Labels and annotations, often produced by third-party or crowd-sourced workers
2. Model layer
- Model architecture and hyperparameters
- Learned weights and parameters (the core intellectual property)
- Pretrained or foundation models obtained from vendors or open repositories
- Fine-tuned derivatives, adapters, and embeddings
- System prompts and guardrail configurations for LLMs
3. Development and training pipeline (MLOps)
- Feature engineering code, notebooks, and experiment tracking
- Training infrastructure (GPUs, cloud compute, orchestration)
- Model registries and artifact repositories
- CI/CD pipelines that package and promote models
- Evaluation and testing harnesses
4. Deployment and inference layer
- Model serving endpoints and APIs
- Inference infrastructure (containers, edge devices, serverless functions)
- Caching, load balancing, and rate limiting components
- Monitoring and logging for drift, performance, and misuse
5. Application and integration layer
- Front-end applications and chat interfaces
- Plugins, tools, function calling, and agent frameworks that let a model take actions
- Downstream systems that consume model outputs (databases, email, payment systems, code execution)
- Orchestration logic that chains multiple models or steps
6. Supply chain
- Open-source libraries and frameworks (PyTorch, TensorFlow, LangChain, etc.)
- Third-party models, datasets, and APIs
- Cloud AI services and managed platforms
- Model cards, licenses, and provenance documentation
7. Human and organizational layer
- Data scientists, ML engineers, and prompt engineers with privileged access
- Business users and end users who provide inputs and act on outputs
- Annotators and contractors
- Governance bodies that approve models for use
What It Is: The Attack Surface
The attack surface is the total set of points where an unauthorized actor can attempt to enter, manipulate, extract data from, or disrupt the system. In AI security, the attack surface is typically organized by lifecycle phase and by the asset being targeted. The exam frequently references two industry taxonomies: the OWASP Top 10 for LLM Applications and MITRE ATLAS (Adversarial Threat Landscape for AI Systems). You should be comfortable with the vocabulary from both.
Attacks on the data layer
- Data poisoning: injecting malicious or mislabeled samples into training or fine-tuning data to degrade performance or implant a backdoor (a trigger that produces attacker-chosen behavior).
- Label flipping: a poisoning variant that changes labels rather than inputs.
- Retrieval poisoning: planting malicious content in documents or web pages that a RAG system will retrieve, enabling indirect prompt injection.
- Data leakage in training sets: sensitive or regulated data (PII, PHI, secrets) included without authorization, later surfaced by the model.
Attacks on the model layer
- Evasion / adversarial examples: crafting inputs with small perturbations that cause misclassification at inference time (e.g., altered images, obfuscated malware, modified text).
- Model extraction / model stealing: querying an API repeatedly to reconstruct a functionally equivalent model or approximate its decision boundary.
- Model inversion: reconstructing representative training inputs (such as faces) from model outputs.
- Membership inference: determining whether a specific record was in the training set, which is a privacy violation.
- Model tampering / weight manipulation: directly altering stored weights or substituting a trojaned model in the registry.
- Backdoored pretrained models: downloading a foundation model that already contains hidden malicious behavior.
Attacks on LLM-specific components
- Direct prompt injection: the user supplies instructions that override the system prompt or safety policies (jailbreaks).
- Indirect prompt injection: instructions hidden in content the model processes (emails, web pages, PDFs, tool outputs) hijack the model without the user intending it.
- System prompt leakage: extracting confidential instructions, credentials, or business logic embedded in prompts.
- Insecure output handling: downstream systems trust model output and execute it (SQL, shell commands, HTML/JavaScript), leading to injection attacks.
- Excessive agency: an agent with too many tools, too much permission, or too much autonomy performs harmful actions when manipulated.
- Sensitive information disclosure: the model reveals memorized training data or data from other users (cross-tenant leakage).
- Unbounded consumption / denial of wallet: attackers trigger expensive queries to exhaust compute budgets or degrade availability.
Attacks on the pipeline and infrastructure
- Compromise of notebooks, CI/CD, or model registries to insert malicious code or artifacts
- Credential theft for cloud AI services
- Insecure deserialization of model files (e.g., pickle-based formats) leading to remote code execution
- Misconfigured storage buckets exposing datasets or weights
- Dependency and plugin vulnerabilities (software supply chain)
Attacks via the human layer
- Insider manipulation of training data or labels
- Social engineering of annotators or engineers
- Over-reliance by users who act on hallucinated or manipulated outputs (automation bias)
How It Works: Mapping Components to Threats and Controls
The practical skill the exam tests is threat modeling an AI system. A reliable method is a four-step approach:
Step 1: Inventory the components. Build an AI asset inventory that lists every model, dataset, pipeline, endpoint, plugin, and third-party dependency. Many organizations fail here because shadow AI (unsanctioned tools) is not captured. Governance frameworks consistently place inventory first.
Step 2: Identify trust boundaries and entry points. Where does untrusted content enter? User prompts, retrieved documents, third-party APIs, uploaded files, training data from external sources, and open-source models are all untrusted inputs. Where do outputs leave and what consumes them?
Step 3: Enumerate threats per component. Use MITRE ATLAS tactics (reconnaissance, resource development, initial access, ML attack staging, exfiltration, impact) or OWASP LLM categories as a checklist. Ask for each component: can it be poisoned, stolen, evaded, manipulated, leaked, or denied?
Step 4: Select layered controls. Map controls to the lifecycle:
- Data controls: provenance tracking, data validation and anomaly detection, access control on datasets, data minimization, differential privacy, sanitization of retrieval sources, content signing.
- Model controls: adversarial training and robustness testing, model watermarking and fingerprinting, encryption of weights at rest, integrity hashes and signed artifacts, secure model formats (e.g., safetensors over pickle), red teaming before release, model cards documenting intended use.
- Inference controls: authentication and authorization on endpoints, rate limiting and query monitoring to detect extraction attempts, input validation and prompt filtering, output filtering and content moderation, confidence thresholds with human review for high-risk decisions, logging of prompts and responses.
- Application and agent controls: least privilege for tools and plugins, human-in-the-loop approval for consequential actions, treating model output as untrusted input to downstream systems, sandboxing code execution, separating system instructions from user content.
- Pipeline controls: secure MLOps (code review, signed commits, protected registries), secrets management, segregation of duties between development and production, vulnerability management for dependencies, software bill of materials extended to an AI bill of materials (AIBOM).
- Supply chain controls: vendor due diligence, evaluation of third-party models and datasets, license review, contractual security requirements, monitoring of upstream advisories.
- Monitoring and response: drift detection, anomaly detection on query patterns, incident response playbooks for AI-specific events (poisoning discovered, jailbreak campaign, model leak), rollback capability to known-good model versions.
A Worked Example
Consider a customer-service chatbot built on a vendor LLM, augmented with RAG over internal policy documents, and able to issue refunds through an API.
- Components: vendor LLM (supply chain), system prompt (model layer), vector database of policy docs (data layer), orchestration code (application layer), refund API (downstream system), customer chat interface (entry point), support staff who upload documents (human layer).
- Attack surface: direct prompt injection by customers; indirect prompt injection via a maliciously edited policy document; excessive agency if the refund tool lacks limits; system prompt leakage revealing business rules; sensitive data disclosure if the vector store contains other customers' data; denial of wallet through long adversarial conversations.
- Priority controls: restrict who can modify the document store and validate uploads; cap refund amounts and require human approval above a threshold; apply output filtering and tool-call validation; isolate tenant data; rate limit sessions; log and monitor for jailbreak patterns; conduct red-team testing before launch.
The exam loves scenarios like this. The correct answer is almost always the control that addresses the specific exposed component at the earliest effective point, with a preference for preventive controls supported by detective monitoring.
Key Distinctions the Exam Expects You to Know
- Poisoning vs. evasion: poisoning occurs at training time and corrupts the model; evasion occurs at inference time and fools a correct model.
- Extraction vs. inversion vs. membership inference: extraction steals the model, inversion reconstructs training data, membership inference confirms whether a record was used.
- Direct vs. indirect prompt injection: direct comes from the user; indirect comes from content the model processes on the user's behalf.
- Insecure output handling vs. excessive agency: the first is about downstream systems trusting output; the second is about the model having too much power to act.
- Attack surface vs. vulnerability vs. threat: attack surface is the set of exposure points; a vulnerability is a specific weakness; a threat is the potential adverse event or actor.
Exam Tips: Answering Questions on AI System Components and Attack Surface
Tip 1: Decompose the scenario before reading the options. Identify which component is being discussed (data, model, pipeline, inference, application, supply chain, human). Most wrong answers are controls for the wrong component.
Tip 2: Pin the lifecycle phase. Ask whether the attack happens during data collection, training, deployment, inference, or operation. A question about corrupted labels is a training-phase issue; the answer will involve data validation or provenance, not output filtering.
Tip 3: Match vocabulary precisely. If the stem describes repeated API queries used to replicate a model, the term is model extraction, and the control is rate limiting plus query monitoring, not adversarial training. ISACA tests terminology carefully.
Tip 4: Treat all external content as untrusted. When a scenario involves documents, web pages, emails, or tool outputs feeding a model, think indirect prompt injection and look for controls on the content source and on tool permissions.
Tip 5: Prefer the earliest preventive control, backed by detection. If options include preventing a poisoned dataset from entering training versus detecting bad model behavior after deployment, prevention earlier in the lifecycle is usually the best answer, unless the question asks for a compensating or detective control.
Tip 6: Remember least privilege for agents. Questions about AI systems taking actions almost always resolve to limiting tool permissions, scoping credentials, and requiring human approval for high-impact actions.
Tip 7: Model output is input to something else. If a downstream system executes or renders model output, the risk is insecure output handling and the control is validation and sandboxing of that output.
Tip 8: Do not forget the supply chain. When a scenario mentions downloading a pretrained model or dataset from a public repository, think integrity verification, provenance, malicious serialization, and vendor due diligence.
Tip 9: Think like a security manager, not just an engineer. AAISM is a management certification. When two options are technically valid, choose the one that aligns with governance: inventory first, risk assessment, documented policies, accountable ownership, and monitoring. Governance and management answers often outrank purely technical fixes when the question asks what should be done first.
Tip 10: Watch for qualifiers. Words like first, most effective, primary, best, and greatest risk change the correct answer. First usually points to inventory, classification, or risk assessment. Most effective points to the control that eliminates the root exposure. Greatest risk points to the component with highest impact and least existing control.
Tip 11: Use frameworks as anchors. Recall OWASP Top 10 for LLM categories and MITRE ATLAS tactics to recognize the threat being described, and recall NIST AI RMF functions (Govern, Map, Measure, Manage) to place the appropriate management response.
Tip 12: Eliminate answers that solve traditional problems only. A firewall or standard web application scanner does not stop data poisoning or prompt injection. If an option is a classic IT control with no AI-specific effect on the described component, it is usually a distractor.
Quick Review Checklist
- Can I list the seven component groups of an AI system?
- Can I name at least two attacks against each component?
- Can I distinguish poisoning, evasion, extraction, inversion, and membership inference?
- Can I explain direct versus indirect prompt injection with an example?
- Can I map at least two controls to each lifecycle phase?
- Can I apply the four-step threat modeling approach to a new scenario in under two minutes?
If you can answer yes to each of these, you are well prepared for AAISM questions on AI System Components and Attack Surface, and you have built the foundation needed for the rest of Domain 3.
Test mode:
AAISM - AI System Components and Attack Surface Example Questions
Test your knowledge of AI System Components and Attack Surface
Question 1
In a retrieval-augmented generation (RAG) architecture, what is the PRIMARY reason the vector database expands the attack surface beyond that of the underlying foundation model?
Question 2
An enterprise is threat modeling an LLM agent that summarizes inbound supplier emails. The agent can also create payment records in the ERP system through a tool API. During red teaming, a crafted supplier email caused the agent to draft an unauthorized payment record. The model provider states that its built-in guardrails block jailbreaks. Which control should the information security manager recommend to BEST reduce the impact of this attack path?
Question 3
An enterprise deploys an internal LLM assistant that uses retrieval-augmented generation. Any employee can upload files to the knowledge source the assistant searches, and the assistant acts on instructions found in retrieved content. While mapping the attack surface, the information security manager must identify the component that MOST enables indirect prompt injection. Which component should be flagged?
Unlock Premium Access
Advanced in AI Security Management
- Access to ALL Certifications: Study for any certification on our platform with one subscription
- 3073 Superior-grade Advanced in AI Security Management practice questions
- Unlimited practice tests across all certifications
- Detailed explanations for every question
- AAISM: 5 full exams plus all other certification exams
- 30-Day Money-Back Guarantee: Full refund within 30 days if unsatisfied
- Risk-Free: 7-day free trial with all premium features!
More AI System Components and Attack Surface questions
21 questions (total)
Other AI Technologies and Controls concepts
AI Architecture Within Enterprise ArchitectureAI Development Life Cycle Security TouchpointsAPI Security for AI ServicesArchitecture Review and Security Sign-Off for AIContent Safety Filters and GuardrailsControl Testing, Assurance and Continuous ValidationCross-Border Data Transfer and Residency for AIData Access Controls and Segregation for AI PipelinesData Classification and Handling for AIData Leakage Prevention for AI Inputs and OutputsData Minimization, Anonymization and PseudonymizationData Provenance, Lineage and Labeling ControlsData Quality, Integrity and Validation ControlsData Retention, Deletion and Unlearning RequestsDetection and Response Integration for AI SystemsEncryption of AI Data at Rest, in Transit and in UseEthical Review and Responsible AI AssessmentExplainability and Interpretability ControlsFairness and Bias Testing ControlsFine-Tuning and Prompt Engineering SecurityIdentity and Access Management for Models and DataInput Validation and Prompt Filtering ControlsKill Switches, Circuit Breakers and Fallback ModesLogging, Auditing and Traceability of AI InteractionsMaintenance, Retraining and Drift ResponseMapping Security Controls to AI RiskMLOps and LLMOps Security ControlsModel Cards, Datasheets and AI DocumentationModel Decommissioning and Secure RetirementModel Selection: Build, Buy, Fine-Tune or PromptModel Validation, Testing and Acceptance CriteriaModel Versioning, Change Management and RollbackMonitoring Model Behavior and Anomaly DetectionOutput Filtering, Moderation and ValidationPre-Deployment Security Testing of AI ModelsPreventive, Detective and Corrective Controls for AIPrivacy by Design for AI SystemsPrivacy-Enhancing Technologies for AIRate Limiting, Quotas and Abuse PreventionRisk-Based Human Oversight of AI Inputs and OutputsRobustness and Reliability ControlsSecrets, Keys and Model Artifact ProtectionSecure Deployment and Release of AI ModelsSecure Deployment: Cloud, On-Premises and Edge AISecure Training Environments and PipelinesSecuring AI Agents, Tools and PluginsSecuring LLM Applications and RAG PipelinesSecurity Architecture Patterns for AI ApplicationsSecurity by Design and Threat-Informed RequirementsSecurity Metrics and Dashboards for AI ControlsSynthetic Data and Its Security ImplicationsTraining Data Security and Integrity ControlsTransparency, Disclosure and User NotificationTrust and Safety Operations for Generative AIZero Trust and Segmentation for AI Workloads