AI-Enabled Attacks: Phishing, Deepfakes and Malware

Why This Topic Matters

The ISACA Advanced in AI Security Management (AAISM) certification expects candidates to understand not only how to secure AI systems, but also how adversaries weaponize AI against organizations. AI-enabled attacks represent a fundamental shift in the threat landscape: attacks that once required skilled human effort, time, and linguistic or technical expertise can now be automated, personalized, and scaled at near-zero marginal cost. For a security manager, this changes risk calculations across the board. Historical controls such as spotting poor grammar in phishing emails, trusting a familiar voice on the phone, or relying on signature-based malware detection are rapidly losing effectiveness.

Within the AI Risk Management for Security domain, this topic sits at the intersection of threat intelligence, risk assessment, control design and governance. Exam questions will test whether you can identify the attack type, understand why AI amplifies it, select the most appropriate control, and connect the threat to governance responsibilities such as risk registers, awareness programs, and incident response planning.

What Are AI-Enabled Attacks?

AI-enabled attacks are cyberattacks in which artificial intelligence, especially generative AI and large language models (LLMs), is used by threat actors to plan, create, automate, personalize or scale malicious activity. It is important to distinguish three related but different concepts that the exam may try to blur together:

1. Attacks using AI (AI as a weapon) - adversaries use AI tools to attack traditional targets. This is the focus of this topic: AI-generated phishing, deepfakes, and AI-assisted malware.
2. Attacks on AI (AI as a target) - adversarial attacks against AI systems themselves, such as prompt injection, data poisoning, model inversion, evasion and model theft.
3. AI as a defense - security teams using AI for detection, response and analytics.

If a question describes attackers using a chatbot to write convincing emails, that is an attack using AI. If a question describes attackers corrupting training data, that is an attack on AI. Keep these categories clear.

Category 1: AI-Enhanced Phishing and Social Engineering

What it is: Phishing is the use of deceptive messages (email, SMS, voice, collaboration platforms) to trick users into revealing credentials, transferring funds, or executing malicious payloads. Generative AI transforms phishing in several ways:

- Flawless language and tone. LLMs eliminate the spelling errors, awkward phrasing and cultural mismatches that users were trained to recognize. Attackers who do not speak the target language can produce native-quality text.
- Hyper-personalization at scale (spear phishing for everyone). AI can scrape LinkedIn, press releases, social media and breach data, then automatically generate tailored messages referencing real projects, colleagues and events. What was once reserved for high-value executives (whaling) can now be done for every employee.
- Business Email Compromise (BEC) enhancement. AI can mimic the writing style of a specific executive after analyzing samples of their emails, making fraudulent payment requests far more convincing.
- Conversational phishing. AI chatbots can conduct multi-turn, real-time conversations with victims, answering questions and overcoming objections, something static phishing templates could never do.
- Multi-channel campaigns. AI orchestrates email, SMS (smishing), voice (vishing) and social media messaging in a coordinated sequence to build credibility.
- Malicious or jailbroken LLMs. Underground tools (often marketed as uncensored models) remove the safety guardrails of mainstream models to write phishing lures and fraud scripts on demand.

Why it matters for risk management: The likelihood component of phishing risk increases sharply because volume, quality and success rates all rise. Controls that depend on user suspicion of poor-quality messages must be re-evaluated.

Category 2: Deepfakes and Synthetic Media

What it is: Deepfakes are synthetic audio, video or images created with deep learning (generative adversarial networks, diffusion models, voice cloning models) that convincingly impersonate real people. In a security context, deepfakes are used for:

- Voice cloning fraud (vishing 2.0). With only a few seconds of recorded audio (from earnings calls, podcasts, voicemail greetings or social media), attackers clone an executive's voice to authorize urgent wire transfers or request credentials. Real-world incidents have resulted in losses of tens of millions of dollars.
- Video deepfakes in real time. Attackers join video conferences as a fabricated CFO or colleague, as in widely reported cases where finance staff transferred funds after a video call populated entirely by deepfake participants.
- Identity verification bypass. Synthetic faces and voices are used to defeat Know Your Customer (KYC) onboarding, liveness checks and biometric authentication.
- Disinformation and reputational attacks. Fabricated videos of executives making offensive statements or announcing false financial news can move markets and damage brands.
- Extortion and harassment. Fabricated compromising media is used to coerce employees into granting access.
- Fake job candidates and insider placement. Deepfaked interviews are used to place fraudulent remote workers inside organizations.

Why it matters for risk management: Deepfakes attack the trust anchors that organizations have relied upon for decades: recognizing a voice, seeing a face, and trusting live interaction. Any process that uses voice or video as implicit authentication is now vulnerable.

Category 3: AI-Assisted and AI-Enabled Malware

What it is: AI lowers the barrier to writing malicious code and increases the sophistication of malware behavior:

- Code generation for low-skill attackers. LLMs can produce functional scripts, exploit code and infostealers, though guardrails and quality limits exist. Script kiddies gain capabilities previously requiring real development skill.
- Polymorphic and metamorphic malware. AI can continuously rewrite malware code while preserving its function, generating unique variants that evade signature-based antivirus and static hash detection.
- Evasion of behavioral detection. Models can learn what sandbox and EDR environments look for and adapt execution to avoid triggering alerts.
- Automated vulnerability discovery and exploit development. AI accelerates reconnaissance, fuzzing, and weaponization of newly disclosed vulnerabilities, shrinking the window between disclosure and exploitation.
- Autonomous and agentic attack chains. AI agents can perform reconnaissance, lateral movement, privilege escalation and data exfiltration with limited human direction, reacting to the environment in real time.
- AI-driven password attacks and CAPTCHA defeat. Models trained on breached password datasets generate highly probable guesses; vision models solve CAPTCHAs that were designed to block bots.
- Supply chain and dependency attacks. Attackers exploit hallucinated package names (slopsquatting) by publishing malicious libraries with names that LLMs commonly invent in code suggestions.

Why it matters for risk management: Detection strategies anchored in known signatures become insufficient. Organizations must shift toward behavioral analytics, zero trust architecture, rapid patching, and assume-breach postures.

How These Attacks Work: The AI-Enabled Kill Chain

Mapping AI to the classic attack lifecycle helps structure both understanding and exam answers:

1. Reconnaissance - AI scrapes and correlates open-source intelligence (OSINT) on targets, building detailed profiles automatically.
2. Weaponization - LLMs craft lures, clone voices, generate deepfake video, and produce polymorphic payloads.
3. Delivery - Automated multi-channel campaigns deliver personalized content at scale, timed to real events (mergers, holidays, leadership changes).
4. Exploitation - Conversational AI handles victim interaction; deepfake calls apply pressure and urgency.
5. Installation and Command-and-Control - Adaptive malware evades detection and adjusts to the environment.
6. Actions on Objectives - Automated data discovery, exfiltration, fraud execution.

The common thread is the collapse of cost, skill and time required at each stage, combined with increased realism and adaptability.

Risk Management Response: What the Exam Expects You to Know

AAISM is a management-level certification. Questions emphasize governance, risk and control selection more than technical implementation. Key responses include:

1. Governance and Risk Assessment
- Update the threat model and risk register to explicitly include AI-enabled attack vectors, with revised likelihood ratings.
- Integrate AI threat intelligence into the risk assessment cycle; monitor the evolving capability of attacker tools.
- Assign ownership for deepfake and AI-fraud risk, typically spanning security, finance, HR, legal and communications.
- Align with frameworks such as the NIST AI Risk Management Framework, ISO/IEC 42001, ISO/IEC 27001 and MITRE ATLAS (adversarial threat landscape for AI systems) and ATT&CK.

2. Process and Procedural Controls (often the BEST answer)
- Out-of-band verification for high-risk transactions: call back on a known number, use a pre-agreed code word, or require in-person or secondary-channel confirmation. This is the single most effective control against voice and video deepfake fraud.
- Segregation of duties and dual authorization for payments and sensitive changes, so that no single deceived individual can complete a fraudulent action.
- Payment verification procedures that do not accept urgency or executive pressure as justification for bypassing controls.
- Never treat voice or video as authentication. Update policies so recognition of a person is not sufficient to authorize action.

3. Technical Controls
- Phishing-resistant multi-factor authentication (FIDO2/WebAuthn, hardware keys) rather than SMS or push-based MFA, since AI-driven social engineering can harvest one-time codes or trigger MFA fatigue.
- Email authentication (SPF, DKIM, DMARC) and AI-powered email security that analyzes intent, context and anomalies rather than keywords alone.
- Behavioral and anomaly-based detection (EDR/XDR, UEBA) to counter polymorphic malware that defeats signatures.
- Zero trust architecture, least privilege and network segmentation to limit the impact of a successful compromise.
- Deepfake detection tools and content provenance standards (such as C2PA) as supporting, not primary, controls. Exam tip: detection tools are useful but unreliable alone; the exam generally favors procedural verification over technology that tries to detect synthetic media.
- Application allowlisting, rapid patching and vulnerability management to counter accelerated exploit development.

4. People Controls
- Modernized security awareness training that stops teaching users to look for typos and instead focuses on verification behavior, urgency red flags, and reporting culture. Include deepfake-specific scenarios and simulations.
- Executive and finance-team targeted training, since these roles are disproportionately impersonated and targeted.
- Reduce the attack surface of public audio and video where feasible, and inform executives that their public media will be used to clone them.

5. Detection and Response
- Update incident response playbooks for deepfake-enabled fraud, BEC, and AI-generated disinformation, including communications and legal steps.
- Establish fast reporting channels so employees who receive suspicious calls or videos can escalate immediately without fear of blame.
- Prepare crisis communications for reputational deepfakes targeting the organization or its leaders.
- Conduct tabletop exercises that include AI-enabled scenarios.

Exam Tips: Answering Questions on AI-Enabled Attacks: Phishing, Deepfakes and Malware

Tip 1 - Identify the attack category first. Read the scenario and classify it: Is AI being used as a weapon (this topic) or is AI the target (adversarial ML)? Is it social engineering (phishing, vishing, deepfake) or technical (malware, exploit)? The correct control depends on the category. Misclassification leads to distractors.

Tip 2 - Recognize what AI changes. When asked why AI increases risk, the expected answers revolve around scale, speed, personalization, realism, lowered skill barrier and adaptability. If an option says AI creates entirely new attack goals, be cautious; AI mostly amplifies existing attack types rather than inventing new objectives.

Tip 3 - For deepfake fraud, process beats technology. When a scenario involves a voice or video request to transfer funds or share credentials, the BEST control is almost always out-of-band verification through a trusted channel combined with dual authorization. Deepfake detection software is a supporting control, not the primary answer.

Tip 4 - For AI phishing, think phishing-resistant MFA plus behavior-based training. If options include SMS-based MFA versus FIDO2 hardware keys, the phishing-resistant option is preferred. If options include training users to spot grammatical errors, that is outdated and likely a distractor.

Tip 5 - For AI malware, signatures are not enough. Polymorphic AI-generated malware defeats signature-based antivirus. Favor answers mentioning behavioral analytics, EDR/XDR, anomaly detection, zero trust and application allowlisting.

Tip 6 - Think like a manager, not an engineer. AAISM questions favor governance actions: updating the risk assessment, revising policies, assigning ownership, aligning to frameworks, and reporting to the board. If two answers seem technically valid, choose the one that addresses root cause and governance over a point solution.

Tip 7 - Watch for the words FIRST, BEST, MOST and PRIMARY. FIRST usually points to assessment or identification (assess risk, classify the incident, verify the request). BEST or MOST effective points to the control that addresses root cause with the broadest coverage. PRIMARY often relates to the main objective or accountability.

Tip 8 - Defense in depth is the expected mindset. Answers claiming a single control eliminates AI-enabled attack risk are usually wrong. Look for layered combinations: process verification + MFA + detection + awareness + response planning.

Tip 9 - Link to incident response and business continuity. Questions may ask what to update after a deepfake incident. Expected answers include playbooks, escalation paths, communication plans, and lessons learned fed back into training and risk registers.

Tip 10 - Know the terminology. Be comfortable with: spear phishing, whaling, BEC, vishing, smishing, voice cloning, synthetic media, liveness detection, polymorphic malware, agentic attacks, MITRE ATLAS, C2PA content provenance, phishing-resistant MFA, out-of-band verification, and zero trust. Recognizing these terms quickly will help you eliminate distractors.

Sample Question Walkthrough

Scenario: A finance employee receives a video call from someone appearing to be the CFO, who urgently instructs an immediate wire transfer to a new vendor account due to a confidential acquisition. Which of the following is the BEST control to prevent loss?

A. Deploy AI-based deepfake detection on video conferencing platforms.
B. Require independent verification through a pre-established channel and dual authorization for new payee transfers.
C. Train employees to notice visual artifacts in video calls.
D. Restrict video conferencing to approved internal platforms.

Analysis: Option B is correct. It is a process control that works regardless of deepfake quality, addresses both the impersonation and the single-point-of-failure risk, and reflects the management mindset AAISM rewards. Option A is a supporting technical control with unreliable accuracy. Option C relies on human detection that improving technology will defeat. Option D does not address impersonation at all.

Key Takeaways

- AI-enabled attacks amplify phishing, deepfakes and malware through scale, personalization, realism and adaptability.
- Deepfakes undermine voice and video as trust anchors; never treat recognition as authentication.
- Polymorphic AI malware defeats signature-based detection; move to behavioral and zero trust approaches.
- Procedural controls such as out-of-band verification and dual authorization are the strongest defenses against AI-enabled fraud.
- Security managers must update risk assessments, policies, training and incident response plans to reflect the AI-enabled threat landscape.
- In the exam, classify the attack, identify what AI changes, prefer process and governance answers, and apply defense in depth.