Security by Design and Threat-Informed Requirements

1. Why This Topic Matters

Artificial intelligence systems are not ordinary software. They learn from data, behave probabilistically, depend on third-party models and libraries, and expose entirely new attack surfaces such as prompt injection, data poisoning, model inversion, and adversarial examples. If security is bolted on after a model has been trained and deployed, the cost of remediation is enormous: retraining models, re-collecting data, re-validating outputs, and in many cases withdrawing a product from the market.

ISACA's AAISM (Advanced in AI Security Management) certification places Security by Design and Threat-Informed Requirements squarely inside the AI Technologies and Controls domain because they represent the shift from reactive to proactive AI security. The exam expects you to understand that security decisions must be made at the point where AI systems are conceived and specified, not after they are built. This topic also connects to the other AAISM domains: governance (who approves the requirements), risk management (which threats justify which controls), and controls (how the requirements are implemented and verified).

In practical terms, organizations that embed security early:
- Reduce the likelihood and impact of AI-specific incidents
- Satisfy regulators (EU AI Act, NIST AI RMF, ISO/IEC 42001, ISO/IEC 27001 Annex A) with demonstrable evidence
- Lower total cost of ownership by avoiding late-stage rework
- Build stakeholder and customer trust in AI outputs

2. What Is Security by Design?

Security by Design (SbD) is the principle that security is a foundational property of a system, deliberately engineered from the earliest stages of the lifecycle (concept, requirements, architecture, data acquisition, model development, deployment, operations, and retirement) rather than added as an afterthought.

Core principles you should know for the exam:

- Secure defaults: Systems ship in their most secure configuration; users must opt out of security, not opt in. For AI, this means restrictive output filters, limited tool permissions, and minimal data retention by default.
- Least privilege: Models, agents, pipelines, and service accounts receive only the access they need. An LLM agent with a database tool should have read-only scoped credentials, not administrative rights.
- Defense in depth: Multiple independent layers (input validation, model guardrails, output filtering, monitoring, human oversight) so that a single failure does not become a breach.
- Fail securely: When a guardrail, API, or classifier fails, the system degrades to a safe state (refuse, escalate to a human) rather than an open state.
- Minimize attack surface: Reduce exposed endpoints, unnecessary plugins, excessive model capabilities, and unneeded training data.
- Separation of duties: Those who build models should not be the sole approvers of their release; data labelers should not control evaluation datasets.
- Privacy by design: Data minimization, purpose limitation, anonymization or differential privacy for training data, and consent management are treated as security requirements.
- Economy of mechanism (simplicity): Simpler architectures are easier to verify; avoid unnecessarily complex agentic chains.
- Complete mediation: Every request to the model or by the model (tool calls, retrieval) is checked against policy every time.
- Auditability and transparency: Logging, provenance, and explainability features are designed in so that security events and model decisions can be reconstructed.

ISACA also emphasizes that SbD is a governance discipline as much as an engineering one. It requires executive sponsorship, defined roles (RACI), policies, and gate reviews that stop insecure AI from progressing to the next lifecycle phase.

3. What Are Threat-Informed Requirements?

Threat-Informed Requirements are security requirements derived explicitly from an analysis of the threats the AI system is likely to face. Instead of a generic checklist, the organization asks: Who would attack this system, how, and with what effect? and then writes testable requirements that address those specific threats.

The process generally follows these steps:

Step 1 - Define the system and its context. Document the AI use case, data flows, trust boundaries, users, integrations, model type (predictive, generative, agentic), deployment model (on-premises, cloud, third-party API), and criticality.

Step 2 - Identify threat actors and their motivations. External attackers, malicious insiders, competitors, nation-states, fraudsters, and even careless legitimate users. Consider capabilities and intent.

Step 3 - Enumerate AI-specific threats. Use recognized frameworks:
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems): tactics and techniques such as reconnaissance of ML artifacts, poisoning training data, evasion attacks, model extraction, and LLM prompt injection.
- OWASP Top 10 for LLM Applications: prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft.
- OWASP Machine Learning Security Top 10: input manipulation, data poisoning, model inversion, membership inference, model theft, supply chain, transfer learning attacks, model skewing, output integrity, and model poisoning.
- STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) applied to each AI component and data flow.
- NIST AI 100-2 (Adversarial Machine Learning taxonomy) and the NIST AI RMF (Govern, Map, Measure, Manage).

Step 4 - Assess and prioritize. Rate likelihood and impact, consider existing controls, and rank threats. Tie this to the enterprise risk appetite and to the AI system's risk classification (for example, high-risk under the EU AI Act).

Step 5 - Derive requirements. Each prioritized threat maps to one or more requirements. Requirements should be specific, measurable, testable, and traceable. Example: Threat = indirect prompt injection via retrieved documents. Requirement = All retrieved content must be treated as untrusted data, isolated from system instructions, scanned for injection patterns, and the model must be prevented from executing tool calls based solely on retrieved content without policy validation.

Step 6 - Validate and verify. Requirements are tested through adversarial testing, AI red teaming, unit and integration tests, and independent assurance. Results feed back into the threat model.

Step 7 - Maintain. Threat models are living documents. Model updates, new data sources, new plugins, or new attacker techniques trigger re-assessment.

4. How It Works Across the AI Lifecycle

Concept and planning: Define business purpose, acceptable use, risk tier, and regulatory obligations. Conduct an initial threat assessment and a privacy impact assessment. Decide whether AI is even the appropriate solution.

Requirements: Produce threat-informed security requirements alongside functional requirements. Define non-functional requirements for robustness, confidentiality of training data, explainability, logging, and human oversight. Establish acceptance criteria.

Data acquisition and preparation: Data provenance, integrity verification (hashing, signing), access control, poisoning detection, bias and quality checks, and data minimization. Secure the data pipeline as a supply chain.

Model design and development: Secure coding, dependency scanning, use of trusted model repositories, verification of pre-trained model integrity (model cards, signatures, SBOM or AI BOM), adversarial training where appropriate, and secrets management.

Evaluation and testing: Adversarial robustness testing, red teaming against the threat model, privacy leakage tests (membership inference), fairness evaluation, and validation that each requirement is satisfied. Security gate review before deployment.

Deployment: Hardened infrastructure, API authentication and rate limiting, input and output guardrails, least-privilege tool access for agents, encrypted storage of model weights, and secure configuration management.

Operations and monitoring: Drift detection, anomaly detection on queries (possible extraction attempts), logging of prompts and outputs with privacy controls, incident response playbooks specific to AI, and periodic re-testing.

Retirement: Secure decommissioning of models and data, revocation of credentials, and documentation retention for audit.

5. Key Supporting Concepts

- Trust boundaries: Points where data moves between differing levels of trust, such as user input entering the model, or model output triggering a tool. Each boundary needs controls.
- Excessive agency: Granting an AI agent more autonomy, permissions, or functionality than necessary. A threat-informed requirement limits scope and inserts human approval for high-impact actions.
- AI supply chain: Pre-trained models, datasets, libraries, plugins, and third-party APIs. SbD requires vendor assessment, integrity verification, and contractual security requirements.
- Shift-left: Moving security activities earlier in the lifecycle. Threat modeling in the requirements phase is the clearest example.
- Traceability matrix: Links each threat to requirements, to controls, to tests, to evidence. Auditors love this and so does ISACA.
- Human-in-the-loop / human oversight: A control that is itself a requirement derived from threats such as hallucination-driven harmful actions or automation bias.
- Guardrails: Technical controls (input classifiers, output filters, content policies, allow-lists for tools) that implement requirements.
- Residual risk acceptance: After controls are applied, remaining risk must be formally accepted by the appropriate owner, not by the development team alone.

6. Common Pitfalls Organizations Make (and the Exam Tests)

- Treating AI as just another application and reusing only traditional web security controls
- Writing generic requirements (the system shall be secure) that cannot be tested
- Performing threat modeling once and never updating it after model or data changes
- Allowing the data science team to self-approve releases
- Ignoring the training data pipeline as part of the attack surface
- Trusting third-party models without verifying integrity or reviewing licensing and data origin
- Relying on a single guardrail instead of defense in depth
- Failing to include business and legal stakeholders when defining acceptable use and risk tolerance

7. How to Answer Exam Questions on This Topic

ISACA exam questions are scenario-based and ask for the BEST, MOST important, FIRST, or PRIMARY answer. Several options may be technically correct; your job is to select the one that reflects ISACA's management-level, risk-based mindset.

Identify the lifecycle phase. If the scenario is in planning or requirements, answers involving threat modeling, defining requirements, or classifying risk usually win. If the scenario is post-deployment, the correct answer may involve updating the threat model or adding compensating controls, but ISACA will often highlight that the root issue was missing early-stage security.

Prefer proactive over reactive. Between adding monitoring after an incident and embedding controls into requirements, Security by Design favors the latter.

Prefer risk-based over checklist-based. If one option says apply all controls from a standard and another says derive controls from a threat assessment aligned to risk appetite, the threat-informed option is typically correct.

Prefer governance and accountability. When the question asks who should approve, define, or accept something, answers involving the business owner, risk owner, or governance committee generally outrank the developer or data scientist.

Look for testability and traceability. A requirement that is specific, measurable, and linked to a threat is better than a vague aspiration.

Watch for AI-specific threats. If the scenario mentions training data, retrieved documents, plugins, agents, or model outputs, think OWASP LLM Top 10 and MITRE ATLAS rather than only firewalls and patching.

8. Exam Tips: Answering Questions on Security by Design and Threat-Informed Requirements

Tip 1 - FIRST step questions: When asked what should be done FIRST when developing a new AI system, the answer is almost always to understand context and risk: define the use case and scope, classify the system's risk level, or perform a threat and risk assessment. Implementing controls, selecting tools, or training the model are never first.

Tip 2 - MOST effective way to reduce AI security costs: Integrate security requirements at the design and requirements phase (shift-left). Rework after deployment is the most expensive option.

Tip 3 - Generic versus threat-informed: If an option describes applying a standard list of controls without reference to the specific system's threats, it is likely a distractor. ISACA favors tailoring controls to identified threats and the organization's risk appetite.

Tip 4 - Framework recognition: Know that MITRE ATLAS is the AI counterpart to MITRE ATT&CK, that OWASP Top 10 for LLM addresses generative AI application risks, and that the NIST AI RMF functions are Govern, Map, Measure, and Manage. Questions may describe a framework without naming it.

Tip 5 - Excessive agency scenarios: If an AI agent can take autonomous actions with real-world impact, the best design requirement is least privilege plus human approval for high-impact actions, not simply better logging.

Tip 6 - Third-party and pre-trained models: The best answer involves verifying provenance and integrity, reviewing documentation (model cards), assessing the vendor, and including security requirements in contracts. Assuming a popular model is safe is a distractor.

Tip 7 - Fail secure: When a guardrail or classifier is unavailable, the correct design choice is to block or route to human review, not to allow the request through to maintain availability, unless the scenario explicitly makes availability the critical requirement (and even then, compensating controls are expected).

Tip 8 - Living threat models: If a scenario describes a change (new data source, model retraining, new plugin, new regulation), the best response is to update the threat model and requirements, then reassess controls. One-time assessments are a red flag.

Tip 9 - Separation of duties and approval: The development team should not be the sole approver of go-live. Look for answers involving independent review, a governance body, or a risk owner accepting residual risk.

Tip 10 - Evidence and traceability: When asked how to demonstrate to auditors or regulators that AI security is adequate, the best answer references traceability from threats to requirements to controls to test results, not merely a statement of policy.

Tip 11 - Eliminate extremes: Options that say never, always, or completely eliminate are usually wrong. Security by Design reduces and manages risk; it does not eliminate it.

Tip 12 - Read for the role: AAISM is a management certification. If an option is deeply technical (tune hyperparameters, change the tokenizer) while another is about requirements, governance, or risk alignment, the management-level option is usually what ISACA wants, unless the question explicitly asks for a technical control.

9. Practice Scenarios

Scenario A: An organization plans to deploy a customer-facing generative AI assistant that can access order databases. Which of the following should be done FIRST?
Options: (a) Implement output filtering, (b) Conduct a threat assessment of the use case and define security requirements, (c) Select a vendor model, (d) Train staff on prompt engineering.
Best answer: (b). Threat-informed requirements come before control selection or technology choices.

Scenario B: After deployment, a chatbot is manipulated through documents in its retrieval store to reveal internal pricing. What is the MOST likely root cause?
Options: (a) Insufficient server patching, (b) Retrieved content was treated as trusted and not isolated from instructions, (c) Lack of user training, (d) Weak password policy.
Best answer: (b). This is indirect prompt injection, an AI-specific threat that a threat-informed requirement would have addressed.

Scenario C: Which approach BEST ensures AI security controls remain appropriate over time?
Options: (a) Annual penetration test, (b) Threat model reviewed and updated whenever the model, data, or integrations change, (c) Applying all controls from an industry checklist, (d) Monitoring uptime.
Best answer: (b). Living threat models tied to change management reflect Security by Design.

Scenario D: Who should accept the residual risk of an AI system after security requirements are implemented?
Options: (a) Lead data scientist, (b) Security engineer, (c) Business or risk owner with appropriate authority, (d) External vendor.
Best answer: (c). Accountability for risk acceptance sits with the business owner, consistent with ISACA governance principles.

10. Summary

Security by Design means building AI systems that are secure from inception through secure defaults, least privilege, defense in depth, fail-secure behavior, and governance gates. Threat-Informed Requirements operationalize this by translating a structured analysis of AI-specific threats (using MITRE ATLAS, OWASP LLM Top 10, STRIDE, and NIST guidance) into specific, testable, traceable requirements that are validated through testing and maintained across the lifecycle. On the AAISM exam, favor answers that are early, proactive, risk-based, tailored to AI-specific threats, governed by accountable owners, and supported by traceable evidence. Master these patterns and this topic becomes one of the most predictable areas of the exam.