c++ server doesn't close TCP socket connection after the connection process is killed on client

Question

I'm encountering a problem which is very strange to me

I have a c++ application(server) deployed on centos and on the client side(also runs on centos), there is a program that will connect the server through timer so that when the number of connections reaches 1k, the timer stops.

I'm able to run the following command to detect connections on the server:

netstat -nat |grep -i "port"| grep "ESTABLISHED"

It works decent up for now. However, after I killed the process from client,There was still a significant number of connections in ESTABLISHED STATUS on the server. And even I shut down the client machine,I was still able to see a lot of active connections in the status of ESTABLISHED on the server after more than 10 hours morning the next day.

Even though there could be packet loss when I killed the process so that it failed to notify the server that TCP connection is closed,I believe there is a default heartbeat(keepalive) mechanism within TCP that is able to check if connection is alive.

Is it reliable to get number of connections through the command mentioned above, otherwise what could be going amiss that the server doesn't release closed connections ?

Answer 1

Keepalive by default is off. You have to enable it, in your case at the server end, per socket.

Answer 2

The default values for TCP keepalive are around 2 hours (in BSD/Linux implementations). Are you sure you have TCP keeaplives options set when you are seeing connectoins still up after 10 hours? I am thinking that your application perhaps is not explicitly setting the keepalive option. One way would be to use get socket option and pass SO_TCPKEEPALIVE to check if the keepalive is indeed set. And if it is not set, then please go ahead and set it.

You might find this discussion helpful: How to use SO_KEEPALIVE option properly to detect that the client at the other end is down?