Cybersecurity Vulnerabilities in AI Systems

Overview
In the PMI Certified Professional in Managing AI (PMI-CPMAI) framework, which follows the CPMAI methodology's iterative phases, the earliest phase is Business Understanding. In the exam outline it appears as Identify Business Needs and Solutions. Here the project team decides whether an AI solution is right for a problem and what risks it carries. One of the most important risks is cybersecurity vulnerability. AI systems inherit every weakness of traditional software. They also add new weaknesses that come from their reliance on data, models and automated decisions. A project manager who ignores these risks at the start may build a solution that cannot be trusted, cannot be deployed, or creates legal and reputational damage.

Why It Is Important
1. Business impact: A compromised AI system can make wrong decisions at scale. Examples include approving fraudulent loans, misdiagnosing patients or leaking customer data. The damage spreads faster than a single human error.
2. Trust and adoption: Users and stakeholders will reject AI they cannot trust. Security is a pillar of Trustworthy AI, alongside fairness, transparency, accountability and privacy.
3. Regulatory compliance: Regulations and frameworks set security expectations for AI. Examples are the EU AI Act, GDPR, HIPAA, the NIST AI Risk Management Framework and ISO/IEC 42001. Non-compliance can stop a project entirely.
4. Cost of late fixes: Security designed in at the Business Understanding phase is far cheaper than retrofitting it after deployment. This is the security by design principle.
5. Go/No-Go decisions: In CPMAI, the feasibility assessment must consider whether the security risks of the AI solution are acceptable. If they cannot be mitigated, the right answer may be to not use AI, or to choose a simpler approach.

What It Is
Cybersecurity vulnerabilities in AI systems are weaknesses in the data, model, infrastructure or human processes of an AI solution that attackers can exploit. They fall into two families.

A. Traditional IT vulnerabilities (still apply to AI):
- Insecure APIs and endpoints
- Weak authentication and access control
- Unpatched software libraries and frameworks
- Misconfigured cloud storage holding training data
- Insider threats
- Insecure network transmission

B. AI-specific vulnerabilities:
- Data poisoning: Attackers inject malicious or mislabeled data into training sets so the model learns wrong or hidden behaviors. It may stay invisible until a specific trigger appears; this is called a backdoor or trojan.
- Adversarial attacks (evasion): Carefully crafted inputs fool a trained model at inference time. Examples are slight pixel changes that make a stop sign read as a speed limit sign, or reworded text that bypasses a spam filter.
- Model inversion: Attackers query a model to reconstruct sensitive training data, such as faces or medical records.
- Membership inference: Attackers determine whether a specific individual's data was used in training. This is a privacy breach.
- Model extraction / model stealing: Repeated queries let an attacker replicate a proprietary model, stealing intellectual property.
- Prompt injection and jailbreaking: These target generative AI and LLMs. Malicious instructions in user input or retrieved content override system instructions, cause data leakage or trigger harmful actions.
- Supply chain vulnerabilities: Pre-trained models, open-source datasets, third-party libraries and vendor APIs may be compromised or contain hidden flaws.
- Data leakage: Models, especially LLMs, can memorize and expose confidential data used in training or fine-tuning.
- Model drift exploitation: Attackers gradually shift input patterns so a continuously learning system degrades or adapts maliciously.
- Excessive agency: AI agents with too many permissions can be manipulated into performing damaging actions in connected systems.
- Denial of service on models: Resource-heavy queries overload expensive inference infrastructure.

How It Works (Attack Surface Across the AI Lifecycle)
Think of vulnerabilities mapped to CPMAI phases.
- Business Understanding: Risks are created when the team fails to classify data sensitivity, define threat models or set security requirements.
- Data Understanding and Data Preparation: Poisoning, unauthorized access, privacy leakage, untrusted data sources and weak data lineage are the main risks.
- Model Development: Compromised pre-trained models, insecure libraries, backdoors and overfitting that enables memorization are the main risks.
- Model Evaluation: Risks arise when testing skips adversarial robustness, red teaming and privacy checks.
- Model Operationalization: Risks include exposed APIs, prompt injection, model extraction, evasion attacks, drift and missing monitoring.

Typical attack flow example: A company fine-tunes a customer service chatbot on internal tickets. The tickets were never scrubbed. An attacker uses prompt injection to make the bot reveal another customer's account details. The root cause lies in the Business Understanding and Data phases: data sensitivity was never assessed and no guardrails were required.

Key Mitigations a Project Manager Should Know
- Threat modeling early in the project, for example with STRIDE or MITRE ATLAS for AI-specific threats.
- Data governance: provenance tracking, lineage, validation, access control, encryption at rest and in transit.
- Privacy-enhancing techniques: anonymization, differential privacy, federated learning and data minimization.
- Adversarial testing and red teaming before deployment.
- Input validation and output filtering (guardrails) for generative AI.
- Least privilege for AI agents and APIs, plus rate limiting to deter extraction and DoS.
- Supply chain vetting of models, datasets and vendors, using tools such as model cards, data sheets and a software or AI bill of materials (SBOM / AI-BOM).
- Continuous monitoring for drift, anomalies and abuse, with incident response plans.
- Human-in-the-loop controls for high-stakes decisions.
- Cross-functional involvement: security, legal, compliance and data owners engaged from the start.
- Alignment with frameworks such as the NIST AI RMF (Govern, Map, Measure, Manage), the OWASP Top 10 for LLM Applications and ISO/IEC 27001 / 42001.

Role of the Project Manager
The PM is not expected to be a security engineer. The PM must:
- Ensure security requirements are identified as part of business needs.
- Include security risks in the risk register and the feasibility / go-no-go assessment.
- Engage security experts and stakeholders early.
- Budget and schedule for security testing.
- Ensure accountability and documentation.
- Make sure the chosen solution's risk level matches the organization's risk appetite.

Exam Tips: Answering Questions on Cybersecurity Vulnerabilities in AI Systems
1. Favor early, proactive action. When asked what the PM should do first, pick the option that identifies and assesses risks early. Good choices are threat modeling, a security assessment or engaging the security team during Business Understanding. Avoid answers that wait for deployment or react only after an incident.
2. Match the attack to its definition. Expect scenario questions. Use these keywords:
- Training data tampering means data poisoning.
- Manipulated input at inference means an adversarial or evasion attack.
- Reconstructing training data means model inversion.
- Was this person in the dataset? means membership inference.
- Copying the model via queries means model extraction.
- Overriding LLM instructions means prompt injection.
- Compromised third-party model or library means a supply chain attack.
3. Choose the root-cause fix over a symptom fix. If data was poisoned, the best answer involves data provenance, validation and governance, not just retraining.
4. Think lifecycle, not one-time. AI security requires continuous monitoring because models drift and threats evolve. Answers suggesting security is done after testing are usually wrong.
5. Remember the business lens. In this domain, questions often ask whether to proceed. If risks are unacceptable and cannot be mitigated, the correct answer may be to reconsider the AI approach or choose a less risky alternative. Do not proceed blindly.
6. Collaboration beats solo action. Prefer answers that involve stakeholders such as security, legal, compliance and data owners. Avoid answers where the PM unilaterally decides technical security matters.
7. Link security to trustworthy AI and compliance. When an option connects security to regulations, privacy and ethical AI frameworks, it is often the strongest choice.
8. Watch for absolute wording. Options with words like eliminate all risk or guarantee are usually wrong. Risk is reduced to an acceptable level, not eliminated.
9. Third-party does not mean risk-free. Using a vendor model or API transfers some responsibility, but the organization remains accountable. Choose answers that include vendor due diligence and contractual security requirements.
10. Privacy and security overlap. Model inversion, membership inference and data leakage are both security and privacy issues. The best mitigations are often data minimization, anonymization or differential privacy.
11. Generative AI questions: Look for guardrails, input/output filtering, least privilege for agents, human review and red teaming as the strongest controls.
12. Eliminate distractors. Remove answers that rely solely on traditional IT controls such as firewalls for AI-specific threats like poisoning or adversarial inputs. Both layers are needed.

Quick Recap
Cybersecurity vulnerabilities in AI combine classic IT weaknesses with new threats to data, models and AI behavior. In the Identify Business Needs and Solutions phase, the project manager must surface these risks, define security requirements, and assess feasibility against risk appetite. The PM must also plan mitigations that span the entire AI lifecycle. On the exam, choose answers that are proactive, collaborative, lifecycle-oriented, root-cause focused and aligned with trustworthy AI principles.