Vulnerability Scanning and Penetration Testing of AI

1. Why This Topic Matters

Artificial intelligence systems are now embedded in business-critical processes: fraud detection, customer service chatbots, code generation, medical triage, credit decisions and security operations themselves. Each of these systems introduces an attack surface that traditional security testing was never designed to find. A conventional vulnerability scanner can tell you that a web server is missing a patch, but it cannot tell you that a large language model (LLM) can be manipulated through a cleverly worded prompt to leak its system instructions, that a training dataset has been poisoned, or that a classifier can be fooled by a handful of altered pixels.

For the ISACA Advanced in AI Security Management (AAISM) certification, this topic sits inside the AI Risk Management for Security domain. The exam expects you to understand how an organisation identifies, validates and prioritises weaknesses in AI systems so that risk can be measured and treated. Vulnerability scanning and penetration testing are the two primary technical assurance activities that convert theoretical AI risk into evidence-based findings. Without them, an AI risk register is built on assumptions rather than facts.

From a governance perspective, these activities matter because:
• Regulators (EU AI Act, NIST AI RMF, ISO/IEC 42001, sector rules such as DORA and HIPAA) increasingly expect demonstrable testing of high-risk AI systems.
• Boards and audit committees want assurance that AI deployments do not create unmanaged exposure.
• AI failures are often silent: a poisoned model keeps producing outputs that look plausible, so proactive testing is the only reliable detection method.
• The AI supply chain (open-source models, third-party APIs, pre-trained weights, plug-ins) imports risk the organisation did not create and cannot see without testing.

2. What It Is: Core Definitions

Vulnerability scanning is the largely automated, repeatable process of examining an AI system and its supporting infrastructure for known weaknesses, misconfigurations and policy deviations. It is broad, frequent and comparatively low cost. It answers the question: What known weaknesses exist in this environment right now?

Penetration testing (pen testing) is a goal-oriented, human-led, authorised simulation of an attack. Testers attempt to exploit weaknesses, chain them together and demonstrate real business impact. It is narrower, deeper, periodic and more expensive. It answers the question: Can an adversary actually achieve a harmful objective against this system, and how?

AI red teaming is a closely related term you will see on the exam. It is a form of adversarial testing that specifically targets the model and its behaviour (bias, harmful content, jailbreaks, data leakage, safety bypasses) rather than only the infrastructure. In many organisations red teaming is treated as a specialised subset of AI penetration testing with a stronger focus on model behaviour and socio-technical harm.

Key distinction to memorise: Scanning identifies potential weaknesses; penetration testing validates and exploits them to prove impact. Scanning is continuous and automated; penetration testing is periodic, scoped and expert-driven. Both feed the risk register and remediation process.

3. What Makes AI Testing Different from Traditional Security Testing

The AAISM exam repeatedly tests whether you understand that AI systems have additional layers that need assessment:

a) The model layer. The trained model itself can be attacked. Key vulnerability classes include:
• Prompt injection (direct and indirect) – malicious instructions embedded in user input or in retrieved content that override intended behaviour.
• Jailbreaking – bypassing safety guardrails to obtain prohibited output.
• Model extraction / theft – querying the model to reconstruct its parameters or decision boundary.
• Model inversion and membership inference – inferring sensitive training data or whether a specific record was used in training.
• Adversarial examples / evasion – crafted inputs that cause misclassification.
• Model denial of service – resource-exhausting inputs that degrade availability or inflate cost.

b) The data layer. Training, fine-tuning and retrieval data can be poisoned, biased, mislabelled or contain embedded secrets. Testing looks for data provenance gaps, unvalidated ingestion pipelines and insecure vector databases used in retrieval-augmented generation (RAG).

c) The pipeline and MLOps layer. Notebooks, experiment trackers, feature stores, model registries, CI/CD for models and orchestration tools often have weak authentication, exposed credentials, insecure deserialisation (for example pickle files) and over-permissive storage buckets.

d) The supply chain layer. Pre-trained models from public hubs, third-party APIs, plug-ins, agents and open-source libraries may carry backdoors, malicious serialised code or licensing and provenance problems. Software bills of materials (SBOMs) and the emerging AI bill of materials (AIBOM) support this testing.

e) The application and integration layer. This is where traditional web, API and cloud testing still applies: authentication, authorisation, rate limiting, input validation, logging, secrets management and excessive agency (models granted more permissions or tool access than needed).

A useful exam heuristic: traditional testing covers the container around the model; AI-specific testing covers the model, the data and the behaviour. A complete programme does both.

4. How It Works: The AI Vulnerability Scanning Process

Step 1 – Inventory and classification. You cannot scan what you do not know exists. Maintain an AI asset inventory (models, datasets, endpoints, pipelines, third-party services) and classify each by risk tier based on data sensitivity, autonomy, business criticality and regulatory exposure. High-risk systems get more frequent and deeper scanning.

Step 2 – Infrastructure and dependency scanning. Run conventional vulnerability scanners against the hosts, containers, Kubernetes clusters, APIs and cloud configurations that serve the AI system. Run software composition analysis (SCA) against ML libraries (for example TensorFlow, PyTorch, scikit-learn, LangChain) for known CVEs. Scan Infrastructure-as-Code for misconfigurations such as public storage buckets holding training data.

Step 3 – Model artefact scanning. Scan serialised model files for malicious code, unsafe deserialisation formats and embedded backdoors. Verify hashes and signatures against the trusted source. Check model cards and licences for provenance.

Step 4 – Data and configuration scanning. Scan datasets for personally identifiable information, secrets and anomalies that suggest poisoning. Review model and application configuration for hard-coded API keys, permissive system prompts, disabled content filters and missing rate limits.

Step 5 – Automated adversarial probing. Use AI-specific scanning tools and frameworks that fire libraries of known jailbreak prompts, injection payloads, toxicity probes and extraction attempts at the model endpoint and report success rates. This is the AI equivalent of a signature-based scanner: it finds known weaknesses quickly but will miss novel ones.

Step 6 – Triage, scoring and reporting. Normalise findings into a consistent severity scheme. Traditional CVSS scores still apply to infrastructure issues; for model-level findings organisations often adapt scoring to reflect likelihood of exploitation, data sensitivity and potential harm. Feed results into the risk register and ticketing systems.

Step 7 – Continuous rescanning. AI systems drift. Models are retrained, prompts are edited, new plug-ins are added. Scanning should be integrated into the MLOps pipeline so that every model promotion triggers a scan, and production endpoints are scanned on a defined cadence.

5. How It Works: The AI Penetration Testing Process

Phase 1 – Scoping and authorisation. Define the target (which models, endpoints, data stores, agents), the objectives (for example exfiltrate training data, bypass content safety, escalate privileges through tool use), the rules of engagement, the testing window, and the constraints (no testing against live customer data, no degradation of production). Obtain written authorisation. For third-party models, confirm the provider terms of service permit testing. This phase is heavily examined because it is where management responsibility is most visible.

Phase 2 – Threat modelling. Map likely adversaries, their capabilities and their goals against the system architecture. Frameworks commonly referenced: MITRE ATLAS (Adversarial Threat Landscape for AI Systems) for AI-specific tactics and techniques; OWASP Top 10 for LLM Applications and OWASP Machine Learning Security Top 10 for vulnerability classes; NIST AI RMF and NIST AI 100-2 (adversarial machine learning taxonomy) for risk language; ISO/IEC 42001 and ISO/IEC 23894 for management system expectations. Threat modelling drives which attacks the testers prioritise.

Phase 3 – Reconnaissance. Testers gather information about the model type, version, system prompt, available tools or plug-ins, data sources, rate limits and error behaviour. Much can be inferred from how the model responds.

Phase 4 – Exploitation. Testers attempt attacks across layers, for example:
• Direct prompt injection to override instructions and reveal the system prompt.
• Indirect prompt injection via a document, web page or email the model will process, causing it to take unauthorised actions.
• Jailbreaks using role-play, encoding, multi-turn manipulation or language switching.
• Data exfiltration through the model, including retrieval of other users or tenants information from a shared vector store.
• Excessive agency abuse, where a model with tool access is coerced into sending emails, executing code or making purchases.
• Evasion of a classifier-based security control (for example a malware or fraud detector) using adversarial inputs.
• Membership inference and model inversion against a prediction API.
• Traditional exploitation of the pipeline: compromised notebook servers, leaked credentials in model registries, insecure deserialisation of model files.

Phase 5 – Post-exploitation and impact demonstration. Good testers show business consequence: what data was exposed, which decisions could be manipulated, what financial or safety impact results. This converts technical findings into risk language executives understand.

Phase 6 – Reporting. The report should contain an executive summary, methodology, findings with severity and evidence, root cause, business impact, and prioritised remediation guidance. For AI findings, root cause frequently points to design issues (trusting model output, missing output validation, over-broad permissions) rather than a single patchable bug.

Phase 7 – Remediation and retest. Fixes may include input and output filtering, stronger system prompts, guardrail models, least-privilege tool permissions, human-in-the-loop controls for high-impact actions, data sanitisation, retraining, rate limiting and architectural isolation. Retesting confirms the fix and checks for regression. Lessons learned update the threat model and the scanning payload library.

6. Governance and Management Considerations (High-Yield for AAISM)

The AAISM is a management certification. Expect questions that ask what the security manager or AI risk owner should do, not how to craft a payload. Key management points:

• Risk-based frequency. Testing cadence should be driven by the AI system risk tier, rate of change and regulatory requirement, not by a fixed calendar alone. Any material change (new model version, new data source, new tool integration) should trigger reassessment.
• Integration into the AI lifecycle. Testing belongs in design (threat modelling), pre-deployment (gate before production), and operations (continuous scanning, periodic pen tests). Shift-left reduces cost.
• Independence. High-risk systems should be tested by parties independent of the development team, and periodically by external specialists to avoid blind spots.
• Third-party and shared responsibility. When using a vendor model or API, the organisation remains accountable for how it is integrated. Contracts should address the right to test, vendor testing evidence, vulnerability disclosure and incident notification.
• Legal and ethical constraints. Testing must respect privacy law, terms of service, and safety. Generating harmful content during red teaming requires controls over who sees it and how it is stored.
• Metrics and reporting. Useful KPIs include percentage of AI assets in inventory and under scanning, mean time to remediate AI findings, jailbreak or injection success rate over time, and number of high-risk systems with a current penetration test.
• Linkage to risk treatment. Findings should map to the risk register, influence risk appetite discussions and inform decisions to accept, mitigate, transfer or avoid risk. A finding that cannot be fixed (an inherent LLM limitation, for example) may require compensating controls and formal risk acceptance by the business owner.
• Documentation for assurance. Keep evidence of scope, authorisation, methodology, results and remediation. Auditors and regulators will ask for it.

7. Common Pitfalls the Exam Likes to Test

• Assuming traditional infrastructure testing is sufficient for AI systems.
• Treating a one-time pre-launch pen test as adequate for a model that is continuously retrained.
• Testing only the model and ignoring the MLOps pipeline, data stores and supply chain.
• Testing in production against real customer data without safeguards.
• Failing to obtain authorisation from the model provider or data owner.
• Reporting findings in purely technical terms without business impact, so they never get prioritised.
• Believing a successful jailbreak can always be fully fixed with a prompt tweak; many model-level issues require layered, defence-in-depth controls.
• Relying on vendor assurances without independent validation for high-risk use cases.

8. Exam Tips: Answering Questions on Vulnerability Scanning and Penetration Testing of AI

Tip 1 – Identify whether the question is about scanning or penetration testing. If the stem mentions automated, frequent, broad coverage, known weaknesses, continuous, the answer is scanning. If it mentions exploitation, proof of impact, authorised simulation, chaining, skilled testers, scoped engagement, the answer is penetration testing or red teaming. Many distractors swap these characteristics.

Tip 2 – Think like a manager, not a hacker. When asked what should be done first or what is most important, the AAISM answer is usually governance-oriented: define scope and obtain authorisation, perform threat modelling, classify the asset, align with risk appetite, or ensure findings are tracked to remediation. Technical actions are rarely the best first step.

Tip 3 – Look for the AI-specific layer. If the scenario describes a chatbot, an LLM agent, a RAG application or a classifier, and one option mentions prompt injection, model extraction, data poisoning, adversarial examples or excessive agency, that option is often the one distinguishing an AI-aware answer from a generic one. Conversely, do not ignore traditional controls; the best answer often combines both.

Tip 4 – Prefer risk-based and lifecycle-integrated answers. Options that say testing should be triggered by risk tier and material change, embedded in the MLOps pipeline, and repeated after retraining are generally stronger than options proposing a fixed annual test or a single pre-launch test.

Tip 5 – Remember shared responsibility and third parties. If the scenario involves a vendor API or open-source model, the correct answer typically keeps accountability with the organisation, requires contractual rights to test or obtain testing evidence, and validates the integration rather than assuming the vendor has covered everything.

Tip 6 – Know the frameworks by purpose. MITRE ATLAS = adversarial tactics and techniques against AI. OWASP Top 10 for LLM Applications = common LLM vulnerability classes (prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plug-in design, excessive agency, overreliance, model theft). NIST AI RMF = govern, map, measure, manage risk functions. ISO/IEC 42001 = AI management system. Questions may ask which framework is most appropriate for a given activity.

Tip 7 – Watch for the word independent. For high-risk systems, independence of the testing function is a recurring theme. If one answer offers developers testing their own model and another offers an independent or external team, favour independence unless the question is specifically about early developer-led shift-left testing.

Tip 8 – Impact beats volume. If asked how to prioritise findings, favour answers that use business impact, exploitability and data sensitivity over answers based solely on raw counts or a single technical score.

Tip 9 – Remediation is layered. If asked how to address a model-level finding such as prompt injection, the best answer is usually defence in depth: input validation, output handling, least-privilege tool access, human approval for high-impact actions, monitoring, and retesting, rather than a single control.

Tip 10 – Eliminate absolutes. Options containing always, never, eliminates all risk, guarantees are almost always wrong. AI testing reduces and makes risk visible; it does not eliminate it. Residual risk must be formally accepted by an accountable owner.

Tip 11 – Read the role in the stem. If the question says you are the AI security manager, choose the option that reflects oversight, prioritisation, communication to stakeholders and assurance. If it says you are the tester, more technical options may be appropriate, but these questions are less common on AAISM.

Tip 12 – Practise the four-step elimination. (1) Identify the layer being discussed (model, data, pipeline, supply chain, application). (2) Identify the activity (scanning vs testing vs red teaming vs remediation vs governance). (3) Remove options that are generic IT answers ignoring AI specifics, or AI answers ignoring governance. (4) Choose the option that is risk-based, lifecycle-integrated, independent where required, and tied to documented remediation.

9. Worked Example Questions

Example 1. An organisation deploys a customer-facing LLM assistant that can issue refunds through an internal API. Which finding from a penetration test represents the greatest risk? Consider options such as outdated TLS on the API gateway, verbose error messages, indirect prompt injection via uploaded documents causing unauthorised refunds, and a missing security header. The best answer is the indirect prompt injection leading to unauthorised refunds because it demonstrates exploitable business impact through excessive agency. The other findings are real but lower impact.

Example 2. A security manager wants to ensure newly retrained fraud models are checked for known weaknesses before each release. Which approach is most appropriate? Options might include an annual external penetration test, automated scanning integrated into the model deployment pipeline, a one-time red team exercise, or relying on vendor attestations. The best answer is automated scanning integrated into the pipeline, because the need is frequent, repeatable, pre-release checking for known issues.

Example 3. Before conducting a red team exercise against a third-party hosted foundation model, what should the security manager do first? Options might include selecting jailbreak payloads, confirming the provider terms permit testing and obtaining authorisation, provisioning test accounts, or scheduling the report readout. The best answer is confirming permission and authorisation, reflecting the governance-first mindset.

Example 4. A penetration test shows that an LLM can be jailbroken with multi-turn prompts despite a hardened system prompt. What is the best management response? Options might include accept the risk because the model cannot be fixed, replace the model immediately, implement layered controls (output filtering, guardrail model, least-privilege tools, monitoring) and formally document residual risk, or disable the chatbot. The best answer is layered controls with documented residual risk, reflecting defence in depth and formal risk treatment.

10. One-Page Summary to Memorise

• Scanning = automated, continuous, broad, finds known weaknesses across infrastructure, dependencies, model artefacts, data, configuration and known adversarial prompts.
• Penetration testing = authorised, scoped, human-led, exploits weaknesses to prove impact across the model, data, pipeline, supply chain and application layers.
• Red teaming = adversarial testing focused on model behaviour and harm.
• AI-specific vulnerability classes: prompt injection, jailbreaks, data poisoning, model extraction, inversion, membership inference, adversarial evasion, model denial of service, insecure plug-ins, excessive agency, supply chain compromise.
• Frameworks: MITRE ATLAS, OWASP LLM Top 10, OWASP ML Top 10, NIST AI RMF and AI 100-2, ISO/IEC 42001 and 23894.
• Management essentials: inventory and classify, risk-based cadence, lifecycle integration, authorisation and legal review, independence for high-risk systems, third-party rights to test, business-impact reporting, remediation tracking, retesting, formal residual risk acceptance.
• Exam mindset: governance first, AI-aware plus traditional, risk-based, layered defence, no absolutes.

Master these distinctions and you will be able to handle the vast majority of AAISM questions on vulnerability scanning and penetration testing of AI with confidence.