TCP troubleshooting

In this section we will learn about different network problems that occur and try to analyze and solve them with lab exercises. Let's start with the Reset (RST) packet.

TCP reset sequence

The TCP RST flag resets the connection. It indicates that the receiver should delete the connection. The receiver deletes the connection based on the sequence number and header information. If a connection doesn't exist on the receiver RST is set, and it can come at any time during the TCP connection lifecycle due to abnormal behavior. Let's take one example: a RST packet is sent after receiving SYN/ACK, as shown in the next image.

RST after SYN-ACK

In this example we will see why RST has been set after SYN-ACK instead of ACK:

Open the RST-01.pcap file in the Wireshark:

As you can see in the preceding figure:

The TCP RST packet should not be seen normally
The TCP RST is set after the first two handshakes are complete. A possible explanation could be one of the following:
- The client connection never existed; a RAW packet was send over the TCP server
- The client aborted its connection
- The sequence number got changed/forged

RST after SYN

This is the most common use case. Open the RST-02-ServerSocket-CLOSED.pcap file in Wireshark. In this example the server was not started, the client attempted to make a connection, and the connection refused an RST packet:

Lab exercise

The steps to generate the RST flag in a generic scenario, when the server is not in the listening state, are as follows:

Open Wireshark, start capturing the packets, and choose display filter tcp.port==8082.
Compile the client program Client0301.java:
```
bash$ ~ javac Client0301.java 
```
Run the client program:
```
bash$ ~ java Client0301
```
View and analyze the RST packet in Wireshark.

TCP CLOSE_WAIT

Often a connection is stuck in the CLOSE_WAIT state. This scenario typically occurs when the receiver is waiting for a connection termination request from the peer.

Tip

To find a socket in the CLOSE_WAIT state, use the following commands:

bash:~ $ netstat -an | grep  CLOSE_WAIT
tcp4       0      0  122.167.127.21.56294    10.0.0.21.9999     CLOSE_WAIT

To demonstrate the CLOSE_WAIT state, open the close_wait.pcap file in Wireshark:

As you can see in the preceding screenshot:

The server closed socket packet#5, set tcp.flags.fin == 1, and set tcp.seq == 2131384057.
The client responded with the ACK packet tcp.ack == 2131384058 in packet#7 and didn't close its socket, which remains in the CLOSE_WAIT state.

CLOSE_WAIT means there is something wrong with the application code, and in the high-traffic environment if CLOSE_WAIT keeps increasing, it can make your application process slow and can crash it.

Lab exercise

The steps to reproduce CLOSE_WAIT are as follows:

Open Wireshark, start capturing the packets, and choose display filter tcp.port==9999.
Compile the Java programs Server0302.java and Client0302.java using the javac command:
```
bash$ ~ javac Server0302.java Client0302.java 
```
Run Server0302 using the java command:
```
bash$ ~ java TCPServer01
```

Verify the server is listening on port 9999:

bash $ netstat -an | grep 999
tcp46      0      0  *.9999                 *.* LISTEN

Run the client program:
```
bash$ ~ java Client0302
```

Check the state of the TCP socket; it will be in the CLOSE_WAIT state:

bash $ netstat -an | grep CLOSE_WAIT
tcp4       0      0  127.0.0.1.56960     127.0.0.1.9999         CLOSE_WAIT

Analyze the packet in Wireshark.

How to resolve TCP CLOSE_STATE

The steps are as follows:

To remove CLOSE_WAIT, a restart is required for the process.
Establishing the FIN packet from both the client and server is required to solve the CLOSE_WAIT problem. Close the client socket and server socket when done with processing the record:
```
socket.close(); à Initiates the FIN flow
```

Open the Client0302.java file and close the socket:

Socket socket = new Socket(InetAddress.getByName("localhost"), 9999);
…
socket.close();
…
Thread.sleep(Integer.MAX_VALUE);

Compile and re-run the Java program. CLOSE_WAIT will not be visible.

TCP TIME_WAIT

The main purpose of the TIME_WAIT state is to close a connection gracefully, when one of ends sits in LAST_ACK or CLOSING retransmitting FIN and one or more of our ACK are lost.

RFC 1122: "When a connection is closed actively, it MUST linger in TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). However, it MAY accept a new SYN from the remote TCP to reopen the connection directly from TIME-WAIT state, if..."

We ignore the conditions because we are in the TIME_WAIT state anyway.

Table of Contents for Packet Analysis with Wireshark

TCP troubleshooting

TCP reset sequence

RST after SYN-ACK

RST after SYN

Lab exercise

TCP CLOSE_WAIT

Tip

Lab exercise

How to resolve TCP CLOSE_STATE

TCP TIME_WAIT

Table of Contents for
Packet Analysis with Wireshark