6.1.7 Lab: Explore Three-way Handshake In Wireshark

Understanding the Three-Way Handshake in Wireshark is essential for anyone learning about network communication. This process forms the foundation of reliable TCP connections, ensuring that data is transmitted accurately between devices. In this article, we will explore how to capture and analyze the three-way handshake using Wireshark, a powerful network protocol analyzer.

The three-way handshake is the method used by TCP to establish a connection between a client and a server. It involves three steps: the client sends a SYN packet, the server responds with a SYN-ACK, and the client finishes with an ACK. This sequence ensures both parties are ready to communicate and agree on initial sequence numbers.

To begin exploring this process in Wireshark, you first need to set up a basic network environment. You can do this by opening a web browser and navigating to a website, or by using a dedicated client-server application. Once your network activity is ready, launch Wireshark and select the appropriate network interface to start capturing packets.

After starting the capture, perform the network action you want to analyze, such as loading a webpage. Stop the capture once the action is complete. Now, you can begin filtering for TCP packets by typing "tcp" in the display filter bar. This will narrow down the packets to only those relevant to your analysis.

Next, locate the initial SYN packet. This is usually the first TCP packet in the capture and is sent by the client to initiate the connection. Right-click on this packet and follow the TCP stream to see the full conversation. In the packet details, you will see the SYN flag set, indicating the start of the handshake.

The second step is the server's response. Look for a packet with both SYN and ACK flags set. This packet acknowledges the client's SYN and also proposes its own initial sequence number. This is the SYN-ACK packet, and it is crucial for the handshake to proceed.

Finally, the client sends an ACK packet to acknowledge the server's SYN-ACK. This completes the three-way handshake, and the connection is now established. In Wireshark, you can verify this by checking the TCP header flags and sequence numbers in each packet.

By analyzing these packets, you can see the exact exchange of information that makes TCP connections reliable. The sequence numbers and acknowledgment numbers ensure that both parties are synchronized and ready for data transfer. This process prevents data loss and ensures that communication is orderly.

Understanding the three-way handshake is not just theoretical; it has practical implications for network troubleshooting and security. For example, if a handshake fails, it can indicate network issues or potential attacks such as SYN flooding. By mastering Wireshark, you can quickly identify and resolve such problems.

In conclusion, exploring the three-way handshake in Wireshark provides valuable insights into how TCP connections are established. By following the steps outlined above, you can capture, filter, and analyze the handshake process, gaining a deeper understanding of network communication. This knowledge is essential for anyone pursuing a career in networking or cybersecurity.

Beyond simply observing the handshake, Wireshark allows for a deeper dive into the nuances of each packet. Examine the source and destination ports – these identify the specific applications communicating. Note the initial sequence numbers (ISNs) generated by both client and server. These random numbers are vital for preventing spoofing and ensuring reliable data ordering. A common practice is to observe the ISN generation; predictable ISNs can be exploited in certain attacks.

Furthermore, you can analyze the TCP options included in the packets. While not always present, these options can contain information like Maximum Segment Size (MSS), Window Scale, and Selective Acknowledgements (SACK). Understanding these options provides insight into how the connection is being negotiated and optimized for performance. For instance, a large MSS indicates the ability to transmit larger chunks of data, potentially improving throughput.

To further enhance your analysis, utilize Wireshark's color coding. TCP packets are often highlighted based on their flags, making it easier to visually identify SYN, SYN-ACK, and ACK packets. You can also customize the color scheme to highlight specific patterns or anomalies. Experiment with different display filters to isolate specific aspects of the handshake, such as packets originating from a particular IP address or port. For example, filtering by ip.src == 192.168.1.100 will show only packets originating from that IP.

Moreover, Wireshark’s “Follow TCP Stream” feature isn’t just limited to the handshake. Once the connection is established, you can use it to observe the entire data exchange, allowing you to see how the sequence and acknowledgment numbers evolve as data is transmitted. This is invaluable for debugging application-level issues and understanding data flow. Consider using the "Statistics -> Conversations" menu to get a summarized view of all TCP conversations captured, sorted by bytes sent or received, which can quickly highlight unusual activity.

In conclusion, exploring the three-way handshake in Wireshark is far more than a simple exercise in packet observation. It’s a gateway to understanding the fundamental principles of reliable network communication. By mastering the techniques of capture, filtering, and detailed packet analysis, you can not only visualize the handshake process but also gain a profound appreciation for the intricacies of TCP and its role in the modern internet. This skillset is invaluable for network administrators, security professionals, and anyone seeking a deeper understanding of how data travels across networks, empowering them to troubleshoot issues, identify vulnerabilities, and ultimately build more robust and secure network environments.

Building on that foundation,you can push your analysis even further by correlating the handshake data with higher‑level protocol activity. For example, once the three‑way handshake is complete, capture the subsequent ACK‑driven flow and compare it against the initial SYN‑packet size and timing. If you notice that the SYN packet is unusually large or arrives from an unexpected source port, it may indicate a probing attempt or a misconfigured application that is inadvertently opening connections on non‑standard ports. In such cases, applying a display filter like tcp.flags.syn == 1 && tcp.analysis.retransmission can instantly surface retransmitted SYNs, pointing to network congestion or a potential DoS‑style attack.

Another powerful technique is to export the captured handshake to a separate file and feed it into tools like tcptrace or Wireshark’s “Export Packet Dissections” feature. This allows you to script automated checks for anomalies such as duplicate SYNs, out‑of‑order SYNs, or SYNs with malformed flags. By integrating these checks into a regular monitoring pipeline, you can catch subtle irregularities before they evolve into larger service disruptions.

When troubleshooting a user‑reported connectivity issue, it is often helpful to isolate the handshake phase from the rest of the traffic. Create a capture filter that only records the first few packets of each TCP flow, for instance tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0. This reduces the amount of data you need to sift through and makes it easier to spot missing or malformed SYN packets. If the capture shows that a client never receives a SYN‑ACK response, the problem may lie with a firewall rule, an overloaded server, or an incorrectly configured Network Address Translation (NAT) device. Conversely, if the server sends multiple SYN‑ACKs for the same client address, it could indicate a spoofed source address attempting to exhaust the server’s half‑open connection table.

Security‑focused analysts often combine handshake inspection with TCP sequence number entropy analysis. By logging the ISNs observed over a period of time, you can calculate the statistical distribution of these numbers. A low‑entropy pattern—where many ISNs cluster around a narrow range—suggests that the implementation may be using predictable sequence generation, a known weakness that attackers can exploit for session hijacking. Wireshark’s built‑in “Statistics → TCP Stream Graph → Round Trip Time” can also reveal irregular RTT spikes that sometimes correlate with handshake‑related delays, helping you pinpoint whether latency issues stem from the initial connection setup or from later data transfer stages.

Finally, remember that the three‑way handshake is just one piece of the TCP puzzle. To truly master network troubleshooting, integrate your handshake insights with other protocol layers: examine the subsequent data payload for checksum errors, verify that the advertised window size remains within expected bounds, and watch for abrupt termination signals such as FIN or RST packets that may indicate an unexpected connection reset. By weaving together these observations, you develop a holistic view of the connection lifecycle, enabling you to diagnose intermittent drops, throughput bottlenecks, and security incidents with far greater precision.

In summary, mastering the three‑way handshake within Wireshark equips you with a diagnostic microscope that reveals the inner workings of TCP connections. From capturing the right packets and applying precise filters, to dissecting SYN and SYN‑ACK behavior, and finally correlating this knowledge with broader protocol metrics, you gain the ability to troubleshoot complex network problems, fortify security postures, and optimize performance. This comprehensive skill set is indispensable for anyone who wants to move beyond surface‑level packet viewing and truly understand how data travels, persists, and interacts across the modern internet infrastructure.