HttpsURLConnection - Intermittent Connection Refused errors

Question

I am connecting to a web service via a java application and the HttpsURLConnection implementation. I open the connection, use try with resource block to open (and close when finished) the input stream and input stream reader, and then call connection.disconnect in the finally block. The code is below and it works most of the time.

    HttpsURLConnection connection = null;
    try{       
         // basic authentication
        String encodedAuthString =
            DatatypeConverter.printBase64Binary((userName + ":" + password)
                .getBytes());   

        connection = (HttpsURLConnection) url.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Authorization", "Basic " + encodedAuthString);

        //try-with-resource to close connections when complete
        try(InputStream in = connection.getInputStream();
            InputStreamReader isr = new InputStreamReader(in)){
            int numCharsRead;
            char[] charArray = new char[1024];
            StringBuffer sb = new StringBuffer();
            while ((numCharsRead = isr.read(charArray)) > 0) {
                sb.append(charArray, 0, numCharsRead);
            }

            jsonResult = sb.toString();
        }

    } 
    catch (Exception e){
        throw e;
    } finally {
        if(connection!= null)
            connection.disconnect();
    }

However, this code could be called several times in a single second by multiple threads (although the actual containing method is synchronized, so only one thread can access the connection at a time). In working with our network guys, we found that even though the application calls "connection.disconnect()" the connection does not disconnect right away on the server. We see the call to disconnect in the server logs, but the java application attempts to connect again before the server has completely disconnected resulting in a "Connection Refused Error".

Is there anyway to force a full disconnect via java before connecting again or via a server setting when disconnect is called? Or a way to make it happen faster so the possibility for errors is reduced? I could retry the connection until it successfully connects, but that seems like a hack.

Can anyone help point me in the right direction?

Update - 01/06/2013 - As I have continued investigating, I have found by reading the tcpdump that my client is connecting on the same port for 100 requests, then on the 101st request, the port changes and I get the error. The connection will eventually recover and start sending and receiving requests on the new port, but it can take up to a minute to recover. I have included the tcpdump below. You can see that when the client sends on the new port, the destination ip returns a "R"eset on the new port. Because of this, I am thinking the error is a result of an error at the destination and not from my client but I am still trying to verify since my understanding at this level is limited.

13:20:25.925760 IP DESTINATION_IP.https > CLIENT_IP.53321: P 197809:197846...
13:20:25.925819 IP CLIENT_IP.53321 > DESTINATION_IP.https: . ack 197846 wi...
13:20:25.926393 IP CLIENT_IP.53321 > DESTINATION_IP.https: P 82874:82911(3...
13:20:25.926424 IP CLIENT_IP.53321 > DESTINATION_IP.https: F 82911:82911(0...
13:20:25.928629 IP CLIENT_IP.53322 > DESTINATION_IP.https: S 702514925:702...
13:20:26.005902 IP DESTINATION_IP.https > CLIENT_IP.53321: . ack 82911 win...
13:20:26.008461 IP DESTINATION_IP.https > CLIENT_IP.53322: R 0:0(0) ack 70...
13:20:26.011274 IP DESTINATION_IP.https > CLIENT_IP.53321: P 197846:197883...
13:20:26.011290 IP CLIENT_IP.53321 > DESTINATION_IP.https: R 494623963:494...
13:20:26.011300 IP DESTINATION_IP.https > CLIENT_IP.53321: F 197883:197883...
13:20:26.011305 IP CLIENT_IP.53321 > DESTINATION_IP.https: R 494623963:494...

Answer 1

The connection is kept alive due to HTTP Keep Alive.

This can be crudely disabled by the server making it impossible for the HTTP client library to determine when it has finished sending data, i.e. sending no Content-Length header and not using chunked encoding either. Many web frameworks do one or the other automatically, however - and the application might rely on them anyway, because the client application still has to determine when the server has finished sending data, and it may rely on something like Content-Length or chunked encoding to determine this!

However, the connection refused error is unlikely to be related to this phenomenon - in fact HTTP Keepalive should make it less likely that you will see a connection refused error, not more, because it reuses existing connections.

Answer 2

Get rid of the disconnect() call. This is preventing HTTP keepalive from working. You are therefore using many more TCP connections, which fills up the backlog queue at the server, which causes connection refusals (on some platforms). Closing the input stream is sufficient in most circumstances.