Cognitive Computing, AI and Extended Reality

Why This Topic Matters
The CSSLP Domain 3 (Secure Software Architecture and Design) requires you to understand how emerging technologies change the security assumptions baked into traditional software designs. Cognitive computing, artificial intelligence (AI), machine learning (ML), and extended reality (XR) are no longer niche; they are embedded in customer service bots, fraud detection, medical imaging, autonomous vehicles, industrial training, and consumer gaming. Each introduces new attack surfaces, new data sensitivities, and new failure modes that classic controls were never designed to address. (ISC)2 includes this topic because a security architect must be able to recognize that a model, a sensor stream, or a headset is part of the system boundary and must be threat modeled, protected, and governed just like a database or an API. If you only know how to secure a web application, you will miss the risks that matter most in modern systems.

What It Is: Key Definitions
Cognitive computing refers to systems that simulate human thought processes, combining machine learning, natural language processing, pattern recognition, and reasoning to interpret unstructured data and support or automate decisions. IBM Watson-style platforms are the classic example. The emphasis is on learning from interaction and adapting over time, which means the system's behavior at deployment may differ from its behavior months later.

Artificial intelligence is the broader discipline of building machines that perform tasks normally requiring human intelligence. Machine learning is the subset where behavior is derived from training data rather than explicitly coded rules. Important sub-categories you should recognize: supervised learning (labeled data), unsupervised learning (pattern discovery), reinforcement learning (reward-based), deep learning (multi-layer neural networks), and generative AI / large language models (LLMs) that produce text, code, images, or audio.

Extended reality (XR) is the umbrella term for virtual reality (VR) (fully immersive synthetic environments), augmented reality (AR) (digital overlays on the physical world, e.g., smartphone AR or smart glasses), and mixed reality (MR) (digital objects that interact with and anchor to the physical environment). XR systems rely on cameras, microphones, depth sensors, eye tracking, hand tracking, inertial sensors, and often cloud rendering and spatial mapping services.

Why These Technologies Break Traditional Security Assumptions
1. Behavior is data-driven, not code-driven. In conventional software, a code review and static analysis can reveal logic flaws. In ML systems, the "logic" lives in model weights derived from data. Flawed, biased, or poisoned data produces flawed behavior that no code review will catch.
2. Non-determinism and opacity. Deep models are often "black boxes." Explainability and auditability, which are core to accountability and non-repudiation, become hard to achieve.
3. The attack surface includes the training pipeline. Data sources, labeling vendors, pre-trained models, open-source libraries, and feature stores are all supply-chain components.
4. Physical and sensory consequences. XR and AI-driven cyber-physical systems can cause real-world harm: a manipulated AR overlay can hide a hazard; a spoofed VR environment can induce disorientation or injury; an adversarial sticker can cause an autonomous vehicle to misread a stop sign.
5. Unprecedented data sensitivity. XR devices capture eye movement, gait, room layouts, bystanders' faces, and biometric responses. AI assistants ingest conversations, documents, and personal context. Much of this is biometric or special-category personal data under privacy law.

How It Works: Threats Specific to AI and Cognitive Systems
You should be able to recognize and name these threats on the exam.

Data poisoning. An attacker injects malicious or mislabeled samples into training data so the model learns a backdoor or degraded behavior. Example: inserting spam emails labeled "legitimate" to weaken a spam filter. Design response: data provenance tracking, trusted data sources, anomaly detection on training sets, data validation gates, and dataset versioning.

Adversarial examples / evasion attacks. Carefully perturbed inputs (often imperceptible to humans) cause misclassification at inference time. Design response: adversarial training, input sanitization and normalization, ensemble models, confidence thresholds with human fallback, and rate limiting to prevent iterative probing.

Model inversion and membership inference. Attackers query the model to reconstruct training data or determine whether a specific individual's record was used in training, which is a confidentiality and privacy breach. Design response: differential privacy, output perturbation, limiting confidence score exposure, query auditing, and data minimization in training.

Model extraction / theft. Repeated API queries allow an attacker to clone a proprietary model. Design response: API authentication, rate limiting, query monitoring, watermarking model outputs, and legal/contractual controls.

Prompt injection (direct and indirect). For LLM-based applications, malicious instructions embedded in user input or in retrieved content (web pages, documents, emails) hijack the model's behavior, exfiltrate data, or trigger unintended tool calls. Design response: treat all model input as untrusted, separate system instructions from user data, constrain tool permissions (least privilege for agents), validate and filter outputs, and never let the model be the sole security control.

Insecure output handling. Model output passed directly to a shell, database, browser, or downstream API enables injection attacks (XSS, SQLi, command injection). Design response: treat model output exactly like untrusted user input: encode, validate, parameterize.

Excessive agency and over-reliance. Granting AI agents broad permissions or letting humans accept AI decisions without verification. Design response: human-in-the-loop for high-impact actions, scoped credentials, approval workflows, and clear accountability.

Supply chain risks. Pre-trained models from public hubs can contain backdoors or malicious serialized code (e.g., unsafe pickle files). Design response: model provenance, signing, scanning, SBOM extended to an "AI BOM" or model card, and sandboxed loading.

Bias, fairness, and ethical failures. Discriminatory outcomes create legal, reputational, and safety risks. Design response: representative datasets, fairness testing, explainability tooling, bias audits, and governance review before deployment.

Sensitive information disclosure and hallucination. Models may memorize and leak secrets or confidently produce false information. Design response: scrub secrets and PII from training data, apply output filtering, ground responses in verified sources (retrieval-augmented generation with access controls), and label AI outputs as such.

How It Works: Threats Specific to Extended Reality
Sensor data and privacy exposure. Always-on cameras and microphones capture bystanders, home layouts, documents on desks, and keystrokes inferred from hand tracking. Eye-tracking data can reveal health conditions and is uniquely identifying. Design response: privacy by design, on-device processing where possible, data minimization, explicit consent and indicators (recording lights), encrypted storage and transport, short retention periods, and classification of biometric data as highly sensitive.

Spatial mapping data. 3D maps of physical spaces are valuable to burglars and stalkers and can reveal secure facility layouts. Design response: treat spatial maps as sensitive, restrict sharing, anonymize, and apply access control.

Immersive deception and social engineering. Attackers can present fake UI elements, counterfeit avatars, or manipulated overlays that users cannot easily distinguish from legitimate ones (sometimes called "perception hacking" or "human joystick" attacks that steer users physically). Design response: authenticated and tamper-evident overlays, trusted UI zones controlled by the platform, avatar identity verification, and safety boundaries (guardian/chaperone systems) that cannot be overridden by applications.

Physical safety. Compromised VR can cause collisions, falls, seizures (flashing content), or motion sickness. Design response: fail-safe defaults, content safety filters, integrity protection for safety-critical functions, and clear separation between application code and safety subsystems.

Device and session security. Headsets are IoT devices with firmware, Bluetooth/Wi-Fi, app stores, and often shared use. Authentication on a headset is awkward, so weak PINs or no locks are common. Design response: secure boot, signed firmware, timely updates, biometric or continuous authentication, per-user profiles, and session timeouts.

Harassment and content integrity. Multi-user virtual environments face impersonation, virtual assault, and abuse. Design response: identity controls, personal boundary features, moderation, logging for accountability, and reporting mechanisms.

Applying Core Secure Design Principles
The exam expects you to map traditional CSSLP principles onto these technologies:
Least privilege: constrain what AI agents and XR apps can access (tools, APIs, sensors). An AR game does not need microphone access.
Defense in depth: never rely on the model alone; wrap it with input validation, output filtering, authorization checks, monitoring, and human review.
Fail secure / fail safe: if the model is uncertain or unavailable, default to a safe state, deny the action, or escalate to a human. For XR, loss of tracking should pause the experience, not continue blindly.
Complete mediation: every inference request and every sensor access should pass through authorization.
Separation of duties: those who curate training data, those who approve model deployment, and those who operate the system should be distinct roles.
Economy of mechanism: prefer simpler, explainable models when they meet requirements; complexity in ML pipelines increases risk.
Psychological acceptability: privacy indicators and safety boundaries must not be so intrusive that users disable them.
Privacy by design and data minimization: collect only sensor and training data needed; anonymize or pseudonymize early.
Zero trust: treat model outputs, retrieved documents, and peer avatars as untrusted until verified.
Secure defaults: ship with recording indicators on, telemetry minimized, and agent permissions off.

Threat Modeling AI and XR Systems
Extend your data flow diagrams to include: training data sources, labeling processes, feature pipelines, model registry, inference endpoints, feedback loops (which can be poisoned), sensors, rendering services, and cloud spatial anchors. Apply STRIDE across these elements: spoofing (fake avatars, spoofed sensor inputs), tampering (poisoned data, altered model weights, manipulated overlays), repudiation (opaque model decisions without logs), information disclosure (model inversion, sensor data leakage), denial of service (resource-exhausting queries, sponge attacks), and elevation of privilege (prompt injection triggering privileged tools). MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is the AI-specific counterpart to ATT&CK and is worth knowing by name.

Relevant Frameworks and Standards
NIST AI Risk Management Framework (AI RMF 1.0) with its four functions: Govern, Map, Measure, Manage, and its characteristics of trustworthy AI (valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, fair with harmful bias managed).
OWASP Top 10 for LLM Applications (prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, model theft).
OWASP Machine Learning Security Top 10 (input manipulation, data poisoning, model inversion, membership inference, model theft, AI supply chain, transfer learning attacks, model skewing, output integrity, model poisoning).
ISO/IEC 42001 (AI management systems) and ISO/IEC 23894 (AI risk management).
EU AI Act risk-tiered regulation (unacceptable, high, limited, minimal risk) with obligations for high-risk systems.
Privacy regulations (GDPR, CCPA/CPRA, BIPA for biometrics) that directly affect XR sensor data and AI training data.
IEEE ethically aligned design guidance and XR safety initiatives.

Lifecycle Considerations (Design Through Operations)
Because CSSLP spans the whole SDLC, remember that architecture decisions must support later phases: build in logging of model inputs/outputs and decisions for auditability; plan for model drift monitoring and retraining governance; design rollback capability for models just as for code; include red-teaming of models and XR experiences in the test strategy; define an incident response path for model compromise (e.g., retrain from clean data, rotate API keys, invalidate cached outputs); and document model cards and data sheets as part of security documentation.

Exam Tips: Answering Questions on Cognitive Computing, AI and Extended Reality
1. Map the new technology to an old principle. Most questions are really testing whether you can apply least privilege, defense in depth, fail safe, input validation, or privacy by design in a new context. When you see "AI agent" think least privilege and human oversight; when you see "model output" think untrusted input; when you see "headset sensor" think data minimization and consent.
2. Know the vocabulary precisely. Be able to distinguish data poisoning (training time, corrupts the model) from adversarial examples (inference time, fools the model), and model inversion (recovers training data) from model extraction (steals the model itself). Distractors often swap these.
3. Training data is part of the attack surface. If a question asks where to apply integrity controls in an ML system and one option mentions the data pipeline or data provenance, it is usually the strongest answer.
4. Never let the model be the only control. Any answer suggesting that a well-trained model alone prevents misuse is wrong. The CSSLP philosophy is layered controls with the model wrapped by validation, authorization, monitoring, and human review for high-impact decisions.
5. Prefer design-phase, preventive, and systemic answers. Because this is Domain 3, the best answer is usually architectural (threat model the pipeline, apply privacy by design, isolate safety functions) rather than operational patching after deployment.
6. For XR, prioritize physical safety and biometric privacy. If an option addresses user physical harm or the protection of eye-tracking/biometric/spatial data, it typically outranks options focused on generic performance or usability.
7. Recognize prompt injection as an injection class. Treat it like SQL injection: the fix is separation of instructions from data, validation, least privilege for downstream actions, and output encoding, not simply "telling the model to ignore malicious instructions."
8. Explainability and accountability matter. Questions about auditability, regulatory compliance, or disputed decisions point toward explainable models, decision logging, and human accountability, consistent with non-repudiation.
9. Watch for supply chain cues. "Downloaded a pre-trained model," "third-party dataset," or "open-source ML library" signals supply chain risk: verify provenance, sign and scan artifacts, sandbox loading.
10. Know the frameworks by name and purpose. NIST AI RMF (Govern/Map/Measure/Manage), OWASP LLM Top 10, OWASP ML Top 10, MITRE ATLAS, ISO/IEC 42001. You are unlikely to need deep detail, but you must recognize which one fits a scenario.
11. Use the (ISC)2 mindset. Choose the answer that protects people first (safety, privacy, fairness), aligns with risk management rather than absolute elimination, and reflects the role of a security architect advising the business, not a developer applying a quick fix.
12. Eliminate absolutes. Options claiming a technique "guarantees" fairness, "completely prevents" adversarial attacks, or "eliminates" privacy risk are almost always distractors.

Worked Example
Scenario: A company is designing a customer support assistant built on a large language model that can look up order history and issue refunds. Which design control BEST reduces the risk of a malicious customer manipulating the assistant into issuing unauthorized refunds?
A. Fine-tune the model on examples of refund fraud so it can recognize manipulation.
B. Require the assistant's refund function to be executed only through an authorization check and human approval above a defined threshold, with the model's permissions limited to read-only order lookup by default.
C. Add a system prompt instructing the model never to issue refunds for suspicious requests.
D. Log all conversations for later fraud investigation.
Analysis: A and C rely on the model itself as the security control, which is unreliable against prompt injection. D is detective, not preventive. B applies least privilege, complete mediation, and human-in-the-loop for high-impact actions, which is the architectural, preventive answer. Correct: B.

Quick Review Checklist
- Cognitive computing = adaptive, learning systems; behavior can drift.
- AI/ML threats: poisoning, adversarial inputs, inversion, membership inference, extraction, prompt injection, insecure output handling, excessive agency, supply chain, bias.
- XR threats: sensor/biometric/spatial data privacy, immersive deception, physical safety, device security, harassment.
- Controls: data provenance, input/output validation, least privilege for agents and apps, human oversight, differential privacy, rate limiting, explainability, privacy by design, safety subsystems isolated from app code.
- Frameworks: NIST AI RMF, OWASP LLM/ML Top 10, MITRE ATLAS, ISO/IEC 42001, EU AI Act.
- Exam approach: translate the novel technology into familiar secure design principles, favor preventive architectural controls, and never trust the model as the sole safeguard.