ChannelClosedException - Upstream Address: Not Available

Question

Our scala/thrift service is using twitter finagle 2.12-18.10.0 with java8. The Service is working fine for single request or handful of requests but when we try to hit the service with lot many requests we are getting

'com.twitter.finagle.ChannelClosedException: ChannelException at remote address: hostname:9091 from service: calcservice-thrift. Remote Info: Upstream Address: Not Available, Upstream id: Not Available, 
    at org.apache.thrift.TApplicationException.readFrom(TApplicationException.java:131)

sample client that hits server (it fails intermittently)

log.info("started");
     IntStream.range(0,500)
             .parallel()
             .forEach(i->{
                 try {
                     //remote call to finagle service
                 }
                 catch(Exception e){
                     log.error("error " ,e);
                 }
             });
     log.info("end");

why does the server randomly closes the channel. what could be the probable fix for it

Answer 1

my guess is that your finagle server config is not designed to handle so many requests in parallel, and many of them simply go unanswered or return with an error message. I've got not experience with such an high number of Waiters, but you could try set this to 501 for a test setup, if you fire 500 client requests in parallel to that server.

ThriftMux.Server()
  .withAdmissionControl
  .concurrencyLimit(
     maxConcurrentRequests = 5
     maxWaiters = 501 
  )
  ...