Apache HttpConnection - What am I doing wrong? Connection leaks?

Question

We have a util class that creates a httpClient. Main and CommentsResource are just simplified examples of how we use the HttpCLient, but the basics are the same. We do create one httpClient per route, naming them to ex "comments", for the comment service. We then get metrics for the specific one.

Due to "NoHttpResponseException" being thrown, I tried to set connectionManager.setValidateAfterInactivity to 100, after some reading on the internetz. However, that made me go empty on connections in the pool instead, quite fast. Seemed that it didnt put back connections to the pool, maybe after a exception is being thrown?

In my metrics I can see that leased increases and increases but are never dropped back to 0. Tried to do a unittest, everything seems to look fine. The leased are dropped to 0.

Anyways, do you see anything wrong with the following code?

[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile

main

public static void main(String[] args) {
//... jersey things
jersey.registerResource(new CommentResource(url, HttpClient.lagHttpClient("comments", metrics, authProvider, 
      connectTimeout, readTimeout, keepAlive)));
///    ...
}

CommentsResource

@Path("comments")
@Produces("application/json")
@Consumes("application/json")
public class CommentResource {

private final String baseUrl;
private final CloseableHttpClient client;

public CommentResource(String baseUrl, CloseableHttpClient client) {
    this.baseUrl = baseUrl;
    this.client = client;
}
@GET
public Object getComments(@QueryParam("form_id") String form_id) {
    return getJson(baseUrl + "/comments?form_id=" + form_id, List.class);
}

private Object getJson(String url, Class<?> aClass) {
    try {
        return client.execute(new HttpGet(url), response -> {
            if (response.getStatusLine().getStatusCode() == 200) {
                return ObjectMapperConfig.objectMapper.readValue(response.getEntity().getContent(), aClass);
            } else if (response.getStatusLine().getStatusCode() == 204) {
                return Response.status(204).build();
            } else if (response.getStatusLine().getStatusCode() == 404) {
                return Response.status(404).build();
            } else {
                throw new RuntimeException("Unexpected response:" + response.getStatusLine());
            }
        });
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}
}

HttpClientUtil

public static CloseableHttpClient createHttpClient(String clientName, MetricRegistry metrics, Supplier<String> authTokenProvider,
                                                int connectTimeout, int readTimeout, int keepAlive) {
    return createHttpClient(clientName, metrics, authTokenProvider, connectTimeout, readTimeout, keepAlive).build();
}

public static HttpClientBuilder createHttpClient(String clientName, MetricRegistry metrics, Supplier<String> authTokenProvider,
                                                     int connectTimeout, int readTimeout, int keepAlive) {

    final InstrumentedHttpClientConnectionManager connectionManager =
      new InstrumentedHttpClientConnectionManager(
        metrics,
        RegistryBuilder.<ConnectionSocketFactory>create()
          .register("http", PlainConnectionSocketFactory.getSocketFactory())
          .register("https", SSLConnectionSocketFactory.getSocketFactory())
          .build(),
        null,
        null,
        SystemDefaultDnsResolver.INSTANCE,
        -1,
        TimeUnit.MILLISECONDS,
        clientName
      );
    connectionManager.setDefaultMaxPerRoute(20);
    connectionManager.setMaxTotal(20);
    connectionManager.setValidateAfterInactivity(20_000); // this has been tested with 100 as well and was actually only added to test to remove "I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to"

    final HttpClientBuilder builder = HttpClientBuilder.create()
      .setRequestExecutor(new InstrumentedHttpRequestExecutor(metrics, METHOD_ONLY, clientName))
      .setConnectionManager(connectionManager);

    if (authTokenProvider != null) {
        builder.addInterceptorFirst((HttpRequest request, HttpContext ctx) ->
          request.addHeader("Authorization", authTokenProvider.get()));
    }
    return builder.addInterceptorLast((HttpRequest request, HttpContext ctx) ->
      request.addHeader("RequestId", MDC.get("RequestId")))
      .setDefaultRequestConfig(RequestConfig.custom()
        .setConnectionRequestTimeout(Integer.getInteger("httpClient_connectionRequestTimeout", 2000))
        .setConnectTimeout(Integer.getInteger("httpClient_connectTimeout", connectTimeout))
        .setSocketTimeout(Integer.getInteger("httpClient_readTimeout", readTimeout)).build())
      .setKeepAliveStrategy((resp, context) -> Integer.getInteger("httpClient_keepAlive", keepAlive));
}

Added a resource to get some info from the connection manager->pools, the clock is now 08:58 and I got 2 leased that are not removed..

{
"leased": [
    {
        "closed": false, 
        "created": "2017-09-19T08:13:02.468", 
        "expiry": "+292278994-08-17T09:12:55.807", 
        "route": "....", 
        "updated": "2017-09-19T08:13:02.468"
    }, 
    {
        "closed": false, 
        "created": "2017-09-19T07:14:01.434", 
        "expiry": "2017-09-19T07:14:06.453", 
        "route": "....", 
        "updated": "2017-09-19T07:14:01.453"
    }
], 
"otherInfo": {
    "defaultMaxPerRoute": 20, 
    "maxTotal": 20, 
    "routes": [
        {
            "hopCount": 1, 
            "layerType": "PLAIN", 
            "layered": false, 
            "secure": false, 
            "targetHost": {
                "hostName": "...", 
                "port": 80, 
                "schemeName": "http"
            }, 
            "tunnelType": "PLAIN", 
            "tunnelled": false
        }
    ], 
    "shutdown": false, 
    "totalStats": {
        "available": 1, 
        "leased": 2, 
        "max": 20, 
        "pending": 0
    }, 
    "validateAfterInactivity": 20000
}
}