Executor Service Threading

Question

I have a REST Webservices processing its request by getting result from the legacy system via socket calling.

Since there are 4 sockets available, I setup a Executorservice with no of 4 threads to queue for such connections. The service initially runs ok, but performance starts to degrade after a while; which I use jconsole and found that it is creating > 3000 threads and finally ended with the following errors:

SEVERE: Servlet.service() for servlet [jersey-serlvet] in context with
path [/DCardServices] threw exception [java.lang.OutOfMemoryError:
unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread

Questions:

1. Why am I getting so many threads created?
2. Do I have to shutdown the Executorservice in this case?? How as I am in a Web service environment.

Following is a snippet of my code.

ExecutorService spool = Executors.newFixedThreadPool(Integer.valueOf(GlobalUtils.getAppProp().getString("socketPoolSize")).intValue());  
ExecutorService executorService = Executors.newFixedThreadPool(4);

public Response queforTdmProcess(JSSigData jsData) {
    return sndWSResponse(jsData.processCardResp1(executorService.submit(new TdmSocketRequest(jsData,socketQueue, spool)).get()));
}

public class TdmSocketRequest implements Callable<String>  {
    Socket s = getSocketFromPool();
    /* connection to socket and get data */
    retSocketToPool(s);
}


public Socket getSocketFromPool() {
    try {
        conSocketConsumer sckconsumer = new conSocketConsumer(getSocketQueue());
        Future<Socket> future = getSpool().submit(sckconsumer); 
        Socket s = future.get();
        System.out.print("Getting socket " +  s.getLocalPort() + " from the pool"  + "\n");

            return s;
        } catch (Exception e) {
            // TODO: handle exception
        }
        return null;
    }

public void retSocketToPool(Socket s) {
        conSocketProducer sckProducer = new conSocketProducer(getSocketQueue(),s);
        System.out.print("Returning socket " +  s.getLocalPort() + " to the pool" + "\n" );
        getSpool().submit(sckProducer);
    }
}

Thanks a great deal in advance for any suggestions and help!

Answer 1

Executors.newFixedThreadPool(Integer.valueOf(GlobalUtils.getAppProp().getString("socketPoolSize")).intValue());

That line causes the JVM to create Integer.valueOf(GlobalUtils.getAppProp().getString("socketPoolSize")).intValue() threads. Socket pools tend to represent how many connections it can create and are usually in the 2000-3000 range. You should not be creating thread pools of that size.

Looking at the rest of your code I am not entirely sure what you are trying to do with that pool. You create sockets in the new threads but then block to retrieve those sockets on the original thread. I suggest you remove all the threading you've added because at the moment it does nothing but add more threads.

Lastly when working with asynchronous processing it is almost always a bad idea to use Future.get() since that negates any benefit from multithreading. Having to use it indicates you need to rethink your design.

Answer 2

1. Why am I getting so many threads created?

ExecutorService spool = Executors.newFixedThreadPool(Integer.valueOf(
GlobalUtils.getAppProp().getString("socketPoolSize")).intValue());

Currently you are creating those many threads and you should not require those many threads.

Check the value of socketPoolSize in your properties and that will answer your query. Check maximum number of processes allowed in your server and that value should be greater than socketPoolSize

Have a look at below question:

"java.lang.OutOfMemoryError : unable to create new native Thread"

2. Do I have to shutdown the Executorservice in this case?? How as I am in a Web service environment.

Yes. You have to shutdown ExecutorService irrespective of your environment. Refer to below SE question:

How to forcefully shutdown java ExecutorService