Inside Linux TCP: From Handshake to Reset or Close

TCP is the backbone of network communication in Linux systems. It’s a connection-oriented protocol that ensures reliable data exchange between a sender and a receiver over a network. Operating at Layer 4 (the Transport Layer) of the OSI model, TCP guarantees that data is delivered in the correct order and without loss.
Understanding TCP is not just theoretical. It’s critical for real-world troubleshooting. For example, when an application fails to connect, or data transfer stalls, knowing how TCP establishes, maintains, and closes connections helps you pinpoint issues like dropped packets, RSTs, or handshake failures. This insight can save hours when diagnosing network problems, firewall misconfigurations, or application-level errors.
In this post, we’ll explore real-world Linux scenarios using tools such as ncat, ss, and tcpdump to observe TCP connections from start to finish, from SYN to FIN or RST. By understanding the basic concepts of TCP, troubleshooting becomes much faster and more effective.

TCP state vs TCP flags:
TCP state: In networking, a TCP state refers to the current condition of a Transmission Control Protocol (TCP) connection. Since TCP is a connection-oriented protocol, it must track whether a connection is being opened, actively transferring data, or being closed. This process is managed by a Finite State Machine (FSM). Both the client and the server move through these states independently based on the packets they send or receive. If you want to observe the TCP state then you can monitor it via ss -at or netstat -tn in a linux.
ESTABLISHED, TIME-WAIT, FIN-WAIT-2 are few states of the TCP.

TCP flags: TCP flags are single-bit control signals in TCP packets that manage the state of a connection, indicating events such as connection setup (SYN), acknowledgment (ACK), termination (FIN), or connection reset (RST). These flags can be seen in packet captures using tcpdump or Wireshark. TCP flags are crucial for network troubleshooting, monitoring, and security because they indicate exactly what a TCP connection is doing at any given moment.
Common TCP Flags are: SYN, ACK, FIN, RST, PSH.

What is TCP Handshake?
A TCP handshake is the process used to establish a reliable TCP connection between a client and a server before any actual data is transferred. It’s also called a three-way handshake because it involves three steps exchanged between the two sides. SYN, SYN-ACK and ACK.

In this example, I will send “hello” data from client(10.32.10.21) to the server(10.32.3.69), which is listening on port 9600. We will analyze the packet flow between the client and the server. We will be using the ncat command on the client side to send the data and tcpdump on the server side to monitor the TCP packets.

For reference:
TCP Flags:

Step 1: Testing TCP Connectivity with ncat

echo "hello" | ncat 10.32.3.69 9600

Tcpdump output on server side.

1766716972.073906 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [S], seq 2335128560, win 64240, options [mss 1460,sackOK,TS val 2523678873 ecr 0,nop,wscale 9], length 0 
1766716972.074026 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [S.], seq 1265705932, ack 2335128561, win 28960, options [mss 1460,sackOK,TS val 3672955275 ecr 2523678873,nop,wscale 7], length 0 
1766716972.074184 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [.], ack 1, win 126, options [nop,nop,TS val 2523678874 ecr 3672955275], length 0

First line, Client initiates TCP connection to port 9600. SYN [S] Flags can be seen as the 1st packet.
Second line, Server acknowledges SYN and sends its own sequence number. SYN+ACK [S.] in 2nd packet.
Third line, Client sends the ACK [.] packet to the server.
Now this completes the 3 way handshake. Connection is now open for data transfer and you will see TCP state as “ESTABLISHED” in the linux box.

If you check through ss command then you will see “ESTAB”. This confirms that the session between client and the server is active.

ss -pnt | grep 9600 
ESTAB 0 0 10.32.3.69:9600 10.32.10.21:39518 users:(("java",pid=33120,fd=84))

Now below line shows how data transfer packet looks like.

1766716972.075239 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [P.], seq 1:7, ack 1, win 126, options [nop,nop,TS val 2523678875 ecr 3672955275], length 6 
1766716972.075288 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [.], ack 7, win 227, options [nop,nop,TS val 3672955276 ecr 2523678875], length 0

First line, Client sends data, [P.] = PSH + ACK push 6 bytes to server immediately. Payload size is 6 bytes.
Second line, Server Ack[.] it and confirms that I have received the 6 bytes (5 bytes for hello and 1 byte for newline) data. By default, echo adds a newline (\n) at the end. Notice the server’s ACK number is now 7. In TCP, the ACK number tells the sender the next sequence number the receiver expects. Since the client sent 6 bytes starting at sequence 1, the server ACKs with 7 (1 + 6 = 7), effectively saying: “I have received everything up to byte 6; please send byte 7 next.”

Usually server will also send data back to the clients and you will see similary kind of packets. This is actually a normal TCP data exchange packet flow between client and the server.

After the data exchange is completed, TCP teardown process will begin.

00:38:40.501716 IP 10.32.10.21.55952 > 10.32.3.69.9600: Flags [F.], seq 7, ack 1151, win 131, options [nop,nop,TS val 2522122312 ecr 3671385878], length 0 
00:38:40.501748 IP 10.32.3.69.9600 > 10.32.10.21.55952: Flags [.], ack 8, win 227, options [nop,nop,TS val 3671398702 ecr 2522122312], length 0
1766716972.075352 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [F.], seq 7, ack 1, win 126, options [nop,nop,TS val 2523678875 ecr 3672955275], length 0 
1766716972.081386 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [F.], seq 1, ack 8, win 227, options [nop,nop,TS val 3672955282 ecr 2523678875], length 0 
1766716972.081531 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [.], ack 2, win 126, options [nop,nop,TS val 2523678881 ecr 3672955282], length 0

First Packet, Client sends FIN + ACK [F.] which means, I have finished sending my data. I want to close my side of the connection. and it also acknowledges all the data received from the server side.
Second Packet, I received your request to close. I’m acknowledging it. Server ACK [.] the client FIN correctly.
Third Packet, Client sends a FIN to the server.
Fourth packet, Server also sends its FIN and ACK to the client.
Fifth packet, The client sends the final ACK.
Now the TCP connection is fully closed. This is a graceful connection close. You can observe a similar TCP teardown packet flow whenever a connection is closed.

Now to test the RST behaviour. Lets stop the service listening on Port 9600 on the server and we send the request on the same port from the client.

echo "hello" | ncat 10.32.3.69 9600
Ncat: Connection refused.

If we do that, the client will receive a “Connection refused” response from the server. If you observe the packet flow, it will look like this or a similar pattern. This is a connection rejection from the server side.

1766895916.006405 IP 10.32.10.21.56518 > 10.32.3.69.9600: Flags [S], seq 3541810289, win 64240, options [mss 1460,sackOK,TS val 2702622704 ecr 0,nop,wscale 9], length 0 
1766895916.006514 IP 10.32.3.69.9600 > 10.32.10.21.56518: Flags [R.], seq 0, ack 3541810290, win 0, length 0

1st Packet, client send a SYN packet.
2nd Packet, RST + ACK. This is the “Connection Refused” signal. The server acknowledges the request but immediately kills it with a Reset.

This tcpdump output shows a connection attempt that was immediately rejected with a TCP RST.

This kind of packet flow analysis can be very helpful during application troubleshooting from the network-level. Usually, if we see RST packets, it may be because the service that should be listening on that port is no longer listening or is not working properly. In such cases, the kernel sends an RST. RST packets can also be generated when the maximum number of connections is reached, causing new connection attempts to be forcefully reset. They may also occur when a service is killed, an incorrect protocol is used, or an RST is sent from the client side. Similarly, RST packets may be sent by firewall rules if such rules are configured.

In some cases, we want to allow connections only from specific networks and block connections from outside. For security reasons, we can configure firewall rules to send RST packets in those scenarios.

iptables -A INPUT -p tcp --dport 9600 -j REJECT --reject-with tcp-reset

This iptables rule will reject incoming request with RST on the port 9600.

The table below shows the different states of TCP communication from the client’s perspective while sending data. It describes which types of packets are sent and received, and what each of them means.

TCP Client State → Client Perspective (Sending Data to Server), Packet Flow (Sequential View)Permalink

When a client initiates a TCP connection and sends data, the connection can go through multiple states:

Handshake

Data Transfer

Connection Teardown

TCP connections often move so fast (milliseconds) that catching every state in real-time can be challenging.