Port Scanning Techniques and Analysis – GIAC GCIH Study Guide
Why Port Scanning Techniques Matter
Port scanning is one of the most fundamental activities in the reconnaissance and enumeration phase of an attack. Understanding port scanning techniques is critical for GCIH candidates because:
- It is one of the first steps an attacker takes to map a target network and identify live hosts, open ports, and running services.
- Defenders must understand scanning techniques to detect, interpret, and respond to scanning activity in logs, IDS alerts, and packet captures.
- The GCIH exam frequently tests your ability to identify scan types from packet captures, flag combinations, and tool output.
- Knowing how scans work helps incident handlers distinguish between benign network activity and malicious reconnaissance.
What Is Port Scanning?
Port scanning is the process of sending specially crafted packets to a target host's TCP or UDP ports to determine which ports are open, closed, or filtered. The goal is to enumerate available services and identify potential attack vectors. Tools like Nmap, Masscan, hping3, and Unicornscan are commonly used for port scanning.
Key TCP Flag Combinations to Know
Understanding TCP flags is essential. The six primary TCP flags are:
- SYN – Synchronize (initiates a connection)
- ACK – Acknowledge
- FIN – Finish (graceful close)
- RST – Reset (abrupt close)
- PSH – Push (send data immediately)
- URG – Urgent
How Each Scanning Technique Works
1. TCP SYN Scan (Half-Open Scan / Stealth Scan)
- Nmap flag: -sS
- Sends a SYN packet to the target port.
- If the port is open: Target responds with SYN/ACK. The scanner sends a RST to tear down the connection before it completes.
- If the port is closed: Target responds with RST/ACK.
- If the port is filtered: No response or an ICMP unreachable message.
- Called "half-open" because the three-way handshake is never completed.
- This is the default Nmap scan type when run with root/administrator privileges.
- Considered stealthy because it does not complete a full connection, so it may not be logged by some applications (though modern IDS/IPS will detect it).
2. TCP Connect Scan (Full Connect Scan)
- Nmap flag: -sT
- Uses the operating system's connect() system call to complete a full TCP three-way handshake (SYN → SYN/ACK → ACK).
- If the port is open: The full handshake completes successfully.
- If the port is closed: The target responds with RST/ACK.
- This is the default scan when Nmap is run without root privileges.
- More easily detected and logged because the connection fully establishes.
- Noisier but more reliable than SYN scans in some environments.
3. TCP FIN Scan
- Nmap flag: -sF
- Sends a packet with only the FIN flag set.
- Per RFC 793, if the port is closed, the target should respond with RST.
- If the port is open, the target should drop the packet silently (no response).
- Does not work reliably against Windows systems (Windows sends RST regardless of port state, violating RFC 793).
4. TCP XMAS Scan
- Nmap flag: -sX
- Sends a packet with FIN, PSH, and URG flags set (the packet is "lit up like a Christmas tree").
- Behavior is the same as a FIN scan: closed ports respond with RST; open/filtered ports give no response.
- Also unreliable against Windows systems.
5. TCP NULL Scan
- Nmap flag: -sN
- Sends a packet with no flags set.
- Closed ports respond with RST; open/filtered ports give no response.
- Same limitations against Windows as FIN and XMAS scans.
6. TCP ACK Scan
- Nmap flag: -sA
- Sends a packet with only the ACK flag set.
- This scan does NOT determine if a port is open or closed. It determines whether a port is filtered or unfiltered.
- If the target responds with RST, the port is unfiltered (a firewall is not blocking it).
- If there is no response or an ICMP unreachable, the port is filtered.
- Used for firewall rule mapping – identifying which ports a stateful firewall is filtering.
7. UDP Scan
- Nmap flag: -sU
- Sends a UDP packet to the target port.
- If the port is closed: Target responds with ICMP Port Unreachable (Type 3, Code 3).
- If the port is open: The target may respond with UDP data, or there may be no response (making open and filtered ports hard to distinguish).
- UDP scanning is significantly slower than TCP scanning because many operating systems rate-limit ICMP unreachable messages.
- Important for finding services like DNS (53), SNMP (161/162), TFTP (69), NTP (123), and DHCP (67/68).
8. TCP Window Scan
- Nmap flag: -sW
- Similar to an ACK scan, but examines the TCP window size field in the RST response.
- Some operating systems return a positive window size for open ports and a zero window size for closed ports in the RST packet.
- Useful for differentiating open from closed ports on certain OS types, but not universally reliable.
9. Idle Scan (Zombie Scan)
- Nmap flag: -sI <zombie_host>
- An extremely stealthy scan that uses a third-party "zombie" host to scan the target.
- Relies on the predictable IP ID (IPID) sequence of the zombie host.
- The attacker never sends packets directly from their IP to the target, making it very difficult to trace.
- Steps: (1) Probe the zombie to note its current IPID. (2) Send a spoofed SYN to the target with the zombie's source IP. (3) If the target port is open, it sends SYN/ACK to the zombie, which responds with RST (incrementing its IPID). If closed, the target sends RST to the zombie, which ignores it (IPID stays the same). (4) Probe the zombie again to check IPID increment.
- IPID increments by 2 = port open; IPID increments by 1 = port closed.
10. Decoy Scan
- Nmap flag: -D <decoy1,decoy2,...>
- Sends scan packets from the attacker's real IP mixed with spoofed packets from decoy IP addresses.
- Makes it harder for the defender to identify the real scanner among multiple apparent sources.
- The attacker's real IP is still in the mix (unless the ME keyword is used to position it).
11. Version Detection and OS Fingerprinting
- -sV: Service/version detection – connects to open ports and analyzes responses to determine the specific service and version running.
- -O: OS fingerprinting – analyzes TCP/IP stack behavior (TTL values, window sizes, DF bit, etc.) to determine the target operating system.
- -A: Aggressive scan – combines OS detection, version detection, script scanning, and traceroute.
Key Nmap Options to Remember
- -p: Specify ports (e.g., -p 1-1024, -p 80,443, -p- for all 65535 ports)
- -Pn: Skip host discovery (treat all hosts as online) – useful when ICMP is blocked
- -sn: Ping scan only (host discovery, no port scan)
- -T0 through -T5: Timing templates (T0=paranoid, T5=insane)
- -oN, -oX, -oG: Output in normal, XML, or grepable format
- -f: Fragment packets to evade firewalls/IDS
- --script: Run NSE (Nmap Scripting Engine) scripts
Detecting and Analyzing Port Scans
As an incident handler, you need to recognize scanning activity:
- IDS/IPS alerts: Snort/Suricata rules can detect SYN scans, XMAS scans, NULL scans, and other anomalous flag combinations.
- Firewall logs: Look for a single source IP hitting many ports on one host (vertical scan) or the same port across many hosts (horizontal/network sweep).
- Packet captures: Use Wireshark or tcpdump to analyze flag combinations. For example, a packet with FIN+PSH+URG flags and no prior established connection is an XMAS scan.
- Connection patterns: Many half-open connections (SYN without completing handshake) from one source suggest a SYN scan.
- ICMP Port Unreachable messages: A burst of these indicates a UDP scan hitting closed ports.
Important Concepts for the Exam
- Know the three-way handshake: SYN → SYN/ACK → ACK
- Understand the difference between open, closed, filtered, and unfiltered port states.
- Remember that FIN, XMAS, and NULL scans do not work reliably on Windows because Windows sends RST for all probes regardless of port state.
- The ACK scan is for firewall mapping, NOT for determining if ports are open.
- UDP scanning relies on ICMP Port Unreachable for closed ports; no response could mean open OR filtered.
- The Idle scan is the stealthiest because the attacker's IP never appears in the target's logs.
- Nmap default scan is SYN scan (-sS) with root privileges; TCP Connect (-sT) without root.
Exam Tips: Answering Questions on Port Scanning Techniques and Analysis
1. Read packet captures carefully. When given a packet capture or flag description, systematically identify the flags set. SYN only = SYN scan. FIN+PSH+URG = XMAS. No flags = NULL. ACK only = ACK scan. Full handshake = Connect scan.
2. Pay attention to the response, not just the probe. Exam questions often describe both the sent packet and the response. A SYN/ACK response to a SYN means the port is open. An RST response to a SYN means the port is closed. No response to a SYN usually means filtered.
3. Know what each scan is used for. If a question asks about determining firewall rules, the answer is ACK scan. If it asks about the stealthiest scan, think Idle scan. If it asks about scanning UDP services, the answer is UDP scan (-sU).
4. Watch for Windows vs. Unix/Linux distinctions. FIN, NULL, and XMAS scans are unreliable against Windows. If a question mentions these scans against a Windows target, the results will likely be inaccurate (all ports may appear closed).
5. Understand the trade-offs. SYN scans are faster and stealthier than Connect scans. UDP scans are slow. Idle scans are stealthiest but require a suitable zombie host with predictable IPID.
6. Remember Nmap's default behaviors. Default port range is the top 1000 ports (not all 65535). Use -p- to scan all ports. Default scan type depends on privileges. -Pn skips ping and is critical when ICMP is blocked.
7. Link tools to techniques. Know that Nmap is the primary tool tested, but also recognize hping3 (for crafting custom packets), Masscan (for high-speed scanning), and Netcat (for banner grabbing and simple port checks).
8. For scenario-based questions, think like both the attacker and the defender. The attacker chooses scan types to avoid detection; the defender analyzes logs and captures to identify the scan type and intent. Match the evidence to the technique.
9. Eliminate wrong answers by checking technical accuracy. For example, if an answer says an XMAS scan has SYN+FIN+URG flags, that's wrong – XMAS is FIN+PSH+URG. Small details like these are commonly tested.
10. Practice with Nmap. Run different scan types in a lab environment and observe the packets in Wireshark. Hands-on experience makes it much easier to answer scenario-based questions quickly and confidently on exam day.
