Malware Analysis Fundamentals

Malware Analysis Fundamentals – A Comprehensive Guide for GIAC GCIH Certification

Introduction to Malware Analysis Fundamentals

Malware analysis is one of the most critical skills in incident response and cyber investigations. As a GCIH (GIAC Certified Incident Handler) candidate, understanding how to analyze malicious software is essential for identifying threats, containing incidents, and preventing future attacks. This guide covers everything you need to know about malware analysis fundamentals for your exam and professional practice.

Why Is Malware Analysis Important?

Malware analysis is important for several key reasons:

1. Incident Response: When an organization is compromised, analysts must quickly determine what the malware does, how it spreads, and what damage it has caused. Without malware analysis skills, responders are essentially working blind.

2. Threat Intelligence: Analyzing malware samples helps build threat intelligence, including indicators of compromise (IOCs) such as file hashes, network signatures, registry modifications, and command-and-control (C2) infrastructure.

3. Containment and Remediation: Understanding malware behavior allows teams to properly contain infections and remediate affected systems. Incomplete analysis may lead to reinfection.

4. Attribution and Investigation: Malware analysis can provide clues about threat actors, their techniques, tactics, and procedures (TTPs), and can support law enforcement investigations.

5. Defense Improvement: By understanding how malware operates, organizations can improve their defensive posture, update detection signatures, and patch vulnerabilities.

What Is Malware Analysis?

Malware analysis is the process of examining malicious software to understand its functionality, origin, and potential impact. It involves dissecting malware samples using a combination of automated tools and manual techniques to extract actionable intelligence.

There are three primary types of malware analysis:

1. Static Analysis
Static analysis involves examining the malware without executing it. This is the safest form of analysis and often serves as the first step.

- Basic Static Analysis: Examining file properties, strings, headers, imports, exports, and metadata. Tools include strings, PEview, PEstudio, file command, and hash generation utilities (md5sum, sha256sum).
- Advanced Static Analysis: Disassembling the binary using tools like IDA Pro, Ghidra, or Radare2 to review the assembly code and understand program logic without running it.

2. Dynamic Analysis
Dynamic analysis involves executing the malware in a controlled environment (sandbox) and observing its behavior.

- Basic Dynamic Analysis: Running the malware and monitoring system changes using tools like Process Monitor (ProcMon), Process Explorer, Regshot, Wireshark, and FakeNet-NG.
- Advanced Dynamic Analysis: Using debuggers such as x64dbg, OllyDbg, or WinDbg to step through code execution, set breakpoints, and analyze malware behavior at the instruction level.

3. Behavioral Analysis
This is a subset of dynamic analysis focused specifically on observing what the malware does at a high level — file system changes, registry modifications, network communications, process creation, and persistence mechanisms.

How Malware Analysis Works – The Process

A structured approach to malware analysis typically follows these steps:

Step 1: Set Up a Safe Analysis Environment
- Use isolated virtual machines (VMs) with snapshots for easy restoration
- Employ tools like REMnux (Linux-based) and FlareVM (Windows-based) as dedicated analysis environments
- Ensure the analysis network is isolated from production networks
- Use INetSim or FakeNet-NG to simulate network services

Step 2: Fingerprinting and Triage
- Generate cryptographic hashes (MD5, SHA1, SHA256) of the sample
- Check hashes against online databases like VirusTotal, MalwareBazaar, or Hybrid Analysis
- Determine the file type using the file command or TrID
- Check for known packers or obfuscation using Detect It Easy (DIE) or PEiD

Step 3: Basic Static Analysis
- Extract strings using the strings utility or FLOSS (FireEye Labs Obfuscated String Solver)
- Examine PE headers, import tables, and export tables
- Look for suspicious API calls (e.g., CreateRemoteThread, VirtualAllocEx, WriteProcessMemory, URLDownloadToFile)
- Identify embedded resources, certificates, or anomalous sections

Step 4: Basic Dynamic Analysis
- Take a snapshot of the VM before execution
- Start monitoring tools (ProcMon, Process Explorer, Wireshark, Regshot)
- Execute the malware and observe:
  • File system changes (files created, modified, deleted)
  • Registry modifications (persistence keys like Run, RunOnce, Services)
  • Network activity (DNS queries, HTTP requests, C2 communications)
  • Process activity (child processes, injection into other processes)
  • Mutex creation (used to prevent multiple instances)

Step 5: Advanced Analysis (As Needed)
- Disassemble or decompile the binary for deeper code review
- Debug the malware to understand encryption routines, unpacking stubs, or anti-analysis techniques
- Analyze obfuscated or encoded payloads

Step 6: Document and Report
- Record all IOCs discovered
- Document the malware's capabilities, persistence mechanisms, and C2 infrastructure
- Create detection signatures (YARA rules, Snort/Suricata rules)
- Share findings with the incident response team and threat intelligence platforms

Key Malware Concepts for the GCIH Exam

Types of Malware:
- Viruses: Self-replicating code that attaches to legitimate programs
- Worms: Self-propagating malware that spreads across networks without user interaction
- Trojans: Malware disguised as legitimate software
- Ransomware: Encrypts files and demands payment for decryption
- Rootkits: Hides malware presence by modifying the OS at a deep level
- Backdoors/RATs: Provide remote access to compromised systems
- Keyloggers: Capture keystrokes to steal credentials
- Droppers/Downloaders: Deliver additional malware payloads
- Fileless Malware: Operates entirely in memory, leaving minimal disk artifacts
- Botnets: Networks of compromised systems controlled by a threat actor

Common Persistence Mechanisms:
- Registry Run keys (HKLM\Software\Microsoft\Windows\CurrentVersion\Run)
- Scheduled tasks and cron jobs
- Windows services
- DLL hijacking and search order abuse
- WMI event subscriptions
- Startup folder entries
- Boot record modifications

Anti-Analysis Techniques:
- Packing and Obfuscation: Compressing or encrypting the binary to hinder static analysis (e.g., UPX, custom packers)
- Anti-VM Detection: Checking for VM artifacts (VMware tools, specific MAC addresses, registry keys) to avoid running in analysis environments
- Anti-Debugging: Using techniques like IsDebuggerPresent, timing checks, or exception-based tricks to detect debuggers
- Code Injection: Injecting malicious code into legitimate processes (process hollowing, DLL injection, reflective DLL injection)
- Encrypted Communications: Using HTTPS, custom encryption, or domain generation algorithms (DGAs) to evade network detection
- String Obfuscation: Encoding strings with XOR, Base64, or custom algorithms to hide IOCs from static analysis

Important Windows API Calls to Recognize:
- CreateProcess / CreateRemoteThread — process creation and injection
- VirtualAlloc / VirtualAllocEx — memory allocation (often for shellcode)
- WriteProcessMemory — writing to another process's memory space
- RegSetValueEx — modifying registry values
- InternetOpen / InternetOpenUrl / URLDownloadToFile — network communications
- GetProcAddress / LoadLibrary — dynamic resolution of API functions
- SetWindowsHookEx — hooking for keylogging or other interception

Indicators of Compromise (IOCs):
- File hashes (MD5, SHA1, SHA256)
- IP addresses and domain names of C2 servers
- File paths and names associated with malware
- Registry key modifications
- Mutex names
- Network signatures (User-Agent strings, URI patterns)
- YARA rules for file-based detection

Essential Tools for Malware Analysis

Static Analysis Tools:
- strings / FLOSS — string extraction
- PEstudio / PEview / PE-bear — PE file analysis
- Detect It Easy (DIE) — packer detection
- IDA Pro / Ghidra — disassembly and decompilation
- YARA — pattern matching for malware identification
- ssdeep — fuzzy hashing for similarity analysis

Dynamic Analysis Tools:
- Process Monitor (ProcMon) — real-time file, registry, and process monitoring
- Process Explorer — detailed process information
- Regshot — registry comparison before and after execution
- Wireshark — network packet capture and analysis
- FakeNet-NG / INetSim — fake network service simulation
- x64dbg / OllyDbg — debuggers for Windows executables
- Cuckoo Sandbox / Any.Run — automated malware analysis sandboxes

Analysis Environments:
- REMnux — Linux distribution for reverse engineering and malware analysis
- FlareVM — Windows-based malware analysis toolkit by Mandiant

Exam Tips: Answering Questions on Malware Analysis Fundamentals

1. Know the Difference Between Static and Dynamic Analysis: The exam frequently tests whether you understand the distinction. Static analysis examines the malware without execution; dynamic analysis involves running the malware and observing behavior. Be clear on which tools belong to which category.

2. Memorize Key Tools and Their Purposes: You should know that ProcMon monitors file/registry/process activity, Wireshark captures network traffic, strings extracts ASCII/Unicode strings, and PEstudio analyzes PE file structure. Questions often present a scenario and ask which tool to use.

3. Understand the Analysis Workflow: Questions may describe a scenario where you receive a suspicious file. Know that the first step is typically fingerprinting (hashing), followed by basic static analysis, then dynamic analysis in a sandbox. You would NOT run an unknown binary on a production system.

4. Recognize Anti-Analysis Techniques: Expect questions about how malware evades analysis — packing, anti-VM checks, anti-debugging, and string obfuscation. Know that detecting VM artifacts or checking IsDebuggerPresent are common evasion techniques.

5. Focus on Persistence Mechanisms: Registry Run keys, scheduled tasks, and services are the most commonly tested persistence mechanisms. If a question asks where malware might establish persistence, think about these locations first.

6. Understand Suspicious API Calls: If a question describes a malware sample importing CreateRemoteThread, VirtualAllocEx, and WriteProcessMemory, recognize this as likely process injection. Know the purpose of common Windows APIs in a malware context.

7. IOC Extraction: Questions may ask what IOCs you can extract from analysis. Remember: hashes, IP addresses, domains, file paths, registry keys, mutex names, and network signatures are all valid IOCs.

8. Safety First — Analysis Environment: If a question asks about best practices, always choose the answer that emphasizes isolated analysis environments, VM snapshots, and network segmentation. Never analyze malware on production systems.

9. Use Process of Elimination: For scenario-based questions, eliminate answers that are clearly wrong (e.g., running malware on a production network, or claiming that static analysis requires execution). Then choose the most appropriate remaining answer.

10. Pay Attention to Exact Wording: The GCIH exam is precise. If a question asks about basic static analysis, it refers to string extraction, hashing, and header examination — not disassembly. If it asks about advanced static analysis, it refers to disassembly and decompilation.

11. Know Malware Types and Their Characteristics: Understand the key differences between worms (self-propagating), trojans (deceptive), ransomware (encryption-based extortion), rootkits (stealth), and RATs (remote access). Questions may describe behavior and ask you to identify the malware type.

12. Practice with Real Scenarios: Many GCIH questions are scenario-based. Practice thinking through analysis workflows: What would you do first? What tool would you use? What does this behavior indicate? The more you practice scenario-based thinking, the better prepared you will be.

13. YARA Rules: Understand that YARA rules are used to identify and classify malware based on textual or binary patterns. Know the basic structure: rule name, strings section, and condition section.

14. Domain Generation Algorithms (DGAs): Understand that some malware uses DGAs to generate pseudo-random domain names for C2 communication, making it harder to block all C2 domains proactively.

15. Index Your Reference Materials: Since the GCIH exam is open book, create a well-organized index of your course materials. Tab sections on malware analysis tools, analysis steps, API calls, persistence mechanisms, and anti-analysis techniques for quick reference during the exam.

Summary

Malware analysis fundamentals are a cornerstone of incident response and the GCIH certification. By understanding the types of analysis (static, dynamic, behavioral), the tools involved, common malware behaviors, persistence mechanisms, anti-analysis techniques, and IOC extraction, you will be well-prepared to answer exam questions and handle real-world malware incidents. Focus on the structured analysis workflow, know your tools and their purposes, and practice scenario-based reasoning to excel on exam day.