Out-of-Memory on Tomcat

Question

I am using Tomcat 7 to deploy my application (using spring controllers) and i see the following exception happening at times on my server. Please help. Do understand that this has something to do with my server memory, but what could be the fact that i have wrong..I am also giving my ulimit details at the end. This happens only at times and not always. No specific repeating scenario or interval of occurrence...

I am using a Red Hat 6 (64 bit) OS The JVM specifics that is set are : -Xms1024m -Xmx1024M -XX:PermSize=64M -XX:MaxPermSize=512m

27 Dec 2013 10:56:15,462 DEBUG [http-bio-5004-exec-41] DispatcherServlet:910 - Could  not complete request
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.atmosphere.util.AtmosphereFilterChain.doFilter(AtmosphereFilterChain.java:171)
    at org.atmosphere.util.AtmosphereFilterChain.invokeFilterChain(AtmosphereFilterChain.java:132)
    at org.atmosphere.handler.ReflectorServletProcessor$FilterChainServletWrapper.service(ReflectorServletProcessor.java:359)
    at org.atmosphere.handler.ReflectorServletProcessor.onRequest(ReflectorServletProcessor.java:204)
    at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:237)
    at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:163)
    at org.atmosphere.container.Servlet30CometSupport.service(Servlet30CometSupport.java:101)
    at org.atmosphere.container.Tomcat7Servlet30SupportWithWebSocket.doService(Tomcat7Servlet30SupportWithWebSocket.java:67)
    at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:95)
    at org.atmosphere.container.Tomcat7Servlet30SupportWithWebSocket.service(Tomcat7Servlet30SupportWithWebSocket.java:62)
    at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1675)
.......

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at sun.net.www.http.KeepAliveCache$1.run(KeepAliveCache.java:112)
    at sun.net.www.http.KeepAliveCache$1.run(KeepAliveCache.java:96)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:95)
    at sun.net.www.http.HttpClient.putInKeepAliveCache(HttpClient.java:353)
    at sun.net.www.http.HttpClient.finished(HttpClient.java:341)
    at sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:100)
    at sun.net.www.MeteredStream.justRead(MeteredStream.java:93)
    at sun.net.www.MeteredStream.read(MeteredStream.java:135)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:112)
    at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
    at org.springframework.web.client.DefaultResponseErrorHandler.getResponseBody(DefaultResponseErrorHandler.java:102)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:87)
    at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:494)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:451)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
    at com.payoda.appviewx.web.adapter.rest.AdapterBase.sendRequest(AdapterBase.java:90)
    at com.payoda.appviewx.web.adapter.rest.DiscoverDeviceAdapter.discoverDeviceByRange(DiscoverDeviceAdapter.java:63)
    at com.payoda.appviewx.web.controller.DeviceController.getDiscoverDeviceUsingRange(DeviceController.java:137)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
    ... 71 more

The details of ulimit are given below

Ulimit details
----------------
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127515
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 20000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 65536
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Answer 1

Create file "setenv.sh" ("setenv.bat") in your $CATALINA_HOME/bin, put the following content into it:

set CATALINA_OPTS="-Xms512m -Xmx1024m"

then restart tomcat.

If it does not solve the problem, try changing parameters "Xms" and "Xmx" to higher values.

Important: don't forget to set executable bit to "setenv.sh" (if you are on Linux).

Answer 2

It has nothing to do with ulimit; JVM imposes its own maximum memory limit. When launching java from command line, you'd use -Xmx10g for 10Gb max memory for JVM, for example. For Tomcat, the place to put the settings depends on which OS you have and how you installed Tomcat.

For example, on Debian with the Tomcat from debian package, you'd typically edit /etc/default/tomcat... and put JAVA_OPTS="-Xmx10g" in it (or however much you want).