very few responses finally DeadlineTimeoutException with httpcomponents-client-5.2.x

Question

I am using https://hc.apache.org/httpcomponents-client-5.2.x/5.2.3/

org.apache.httpcomponents.client5:httpclient5:5.2.3 

to make REST API calls which require high throughput.

Java version: 17.0.8, vendor: Red Hat, Inc., runtime: D:\unzipped\java-17-openjdk-17.0.8.0.7-1.win.x86_64

To test the I have create a small Spring REST controller which is an echo server. Using threads, I am firing concurrent requests with 1 millisecond delay inside each thread between two consecutive requests.

Initially I am getting very few responses compared to the number requests (vide a few samples below):

 Thu Mar 14 00:12:44 IST 2024 Stats-sent:10
 Thu Mar 14 00:12:44 IST 2024 Stats-success:0
 Thu Mar 14 00:12:44 IST 2024 Stats-failure:0
 totalStats [leased: 0; pending: 0; available: 0; max: 5000]
 Number of routes []
 Thu Mar 14 00:12:49 IST 2024 Stats-sent:8740
 Thu Mar 14 00:12:49 IST 2024 Stats-success:192
 Thu Mar 14 00:12:49 IST 2024 Stats-failure:0
 totalStats [leased: 5; pending: 8517; available: 0; max: 5000]
 Number of routes [{}->http://172.18.17.74:8093]
 Thu Mar 14 00:12:54 IST 2024 Stats-sent:12818
 Thu Mar 14 00:12:54 IST 2024 Stats-success:432
 Thu Mar 14 00:12:54 IST 2024 Stats-failure:0
 totalStats [leased: 5; pending: 12359; available: 0; max: 5000]
 Number of routes [{}->http://172.18.17.74:8093]
 Thu Mar 14 00:12:59 IST 2024 Stats-sent:15900
 Thu Mar 14 00:12:59 IST 2024 Stats-success:660
 Thu Mar 14 00:12:59 IST 2024 Stats-failure:0
 totalStats [leased: 5; pending: 15204; available: 0; max: 5000]

Then after sometime, I am continuously getting the following exception:

DeadlineTimeoutException: Deadline: 2024-03-13T17:46:18.758+0000, -10847 MILLISECONDS overdue

I have chosen this version as this very exception has been resolved from this version: See https://issues.apache.org/jira/browse/HTTPCORE-754

A few relevant configuration is supplied below:

   final int MAX_CONN_PER_ROUTE = Integer.getInteger("max.conn.per.route", 500);
   final int MAX_CONN_TOTAL = Integer.parseInt(System.getProperty("max.conn.total", "5000"));

   final Resolver<HttpRoute, ConnectionConfig> connectionConfigResolver = route -> { 
                    return ConnectionConfig.custom().setConnectTimeout(Timeout.ofSeconds(2 * 60))
                .setSocketTimeout(Timeout.ofSeconds(2 * 60)).setValidateAfterInactivity(Timeout.ofSeconds(1))
                .setTimeToLive( TimeValue.of(1, TimeUnit.MINUTES)).build();
    };


   final PoolingAsyncClientConnectionManager cm = PoolingAsyncClientConnectionManagerBuilder.create() .setConnectionConfigResolver(connectionConfigResolver).setTlsConfigResolver(tlsConfigResolver).setMaxConnPerRoute(MAX_CONN_PER_ROUTE).setMaxConnTotal(MAX_CONN_TOTAL).build();
    cm.setDefaultMaxPerRoute(Integer.getInteger("client5.PoolingMaxConnectionSize", 5));
    final HttpAsyncClientBuilder clientBuilder = HttpAsyncClients.custom().setConnectionManager(cm);
    final IOReactorConfig ioReactor = IOReactorConfig.DEFAULT;
    IOReactorConfig.Builder builder = IOReactorConfig.copy(ioReactor);

    final Integer count = Integer.getInteger("client5.IOThreadCount", Runtime.getRuntime().availableProcessors());
    builder.setIoThreadCount(count);
    builder.setSelectInterval(TimeValue.of(2, TimeUnit.MILLISECONDS));
    //      builder.setSoReuseAddress(true);
    clientBuilder.setIOReactorConfig(builder.build());

    CloseableHttpAsyncClient closeableHttpAsyncClient = clientBuilder.build();
    closeableHttpAsyncClient.start(); 

I will be highly obliged if somebody can throw some light on this or pinpoint my mistake.