- OS: Windows Server 2008 R2 SP1
- Web Server: IIS 7.5
- JSP/Servlet Engine: Tomcat 5.5.28 (32-bit)
- PHP: 5.4.14
- Java: JRE SE 1.6.0_20 (32-bit)
- Apache Isapi Connector hooks into Tomcat from IIS
- PHP-Java Bridge 6.2.1
- BMC AR System 7.5 Patch 6
- Tomcat Initial and Max Memory: 1024 MB, 1024 MB
I am using a Java web application called AR System. After installing the PHP-Java Bridge, I started seeing java.lang.OutOfMemoryError: PermGen space error in the Tomcat logs. (I see in Windows Task Manager that there are 6 PHP-CGI.exe processes, all similar in memory footprint, give or take 5 MB). It would occur every other day or so and then shortened to every day, sometimes twice a day. Consequently, the application hangs and I have to restart it. And I added a Windows Task to restart Tomcat during non-peak hours to give me some cushion. I suspected a memory leak and started doing some research. Normally, Tomcat sits at around 300-350 MB. With the PHP-Java Bridge, memory jumped up significantly. In fact, the error has occurred anywhere from 450-600 MB.
I learned that default PermGen is 64MB and PermGen should be set to 1/4, up to 1/3 of Tomcat memory (sorry, I don't recall the link). Tomcat is running under Windows Services at this point, and I added the following to its properties:
- -XX:+UseConcMarkSweepGC
- -XX:+CMSPermGenSweepingEnabled
- -XX:+CMSClassUnloadingEnabled
- -XX:PermSize=128M
- -XX:MaxPermSize=256M
I enforced GC on PermGen memory and increased the size from the default 64 MB size to 128-256 MB. Memory went up all the way to 800-850 MB, slowly, but it wasn't hanging during peak hours, albeit I still had Tomcat intentionally restart during non-peak hours, via a Windows Task. If I take off the restart, it MIGHT eventually hang but I haven't tried it.
I still suspected a memory leak. I installed a trial version of AppDynamics to monitor the application, its memory, and run leak detection. Additionally, to use tools like VisualVM and Memory Analyzer (MAT), I disabled The Tomcat Windows service and ran Tomcat from the Windows Command Line, via catalina.bat. I appended Java Options to the file; I made sure Tomcat memory was 1024 MB, Perm Gen was 128/256 MB, and ensured PHP-Java Bridge and AppDynamics was running. As of right now, PermGen is holding at 163 MB used, and AppDynamic's Automatic Leak Detection did not detect any leaks with any Java Collections.
I fired up MAT, created a heap dump and analyzed for leaks. When I ran it yesterday, it found three possible suspects:
- net.sf.ehcache.Cache
- net.sf.ehcache.store.DiskStore
- org.apache.catalina.loader.WebappClassLoader
When I ran it today, it found 2 possible suspects:
- java.util.HashMap
- org.apache.jasper.servlet.JasperLoader
So, with MAT and AppDynamics, it appears that no memory leaks were detected for classes directly related to the PHP-Java Bridge JAR files. I haven't tried using Plumbr, but I can't find the free beta version. The free version detects leaks, but you have to pay to see it.
Again, I don't have a source link at this time, but I recall reading that Tomcat 5.x can have performance and memory leak issues. Of course, that doesn't mean everybody will have those issues, just a select number. I know Tomcat 6 and Tomcat 7 redesigned their memory management or how they structure memory. I also did speak with someone from BMC, the maker of AR System, and they said the current version of AR System I'm using could suffer from performance and memory issues. But, again, none of this was a problem before the PHP-Java Bridge. It was only after I installed it that this PermGen memory issue started.
Since the tools above did not report any leaks, does that mean there are no leaks and PHP-Java Bridge just needed more than 64 MB PermGen memory? Or, is there an inherit problem with my version of Tomcat and installing the PHP-Java Bridge just broke the proverbial camel's back?
Upgrading to a newer version of AR System and Tomcat is not an option. If there is a leak, I can uninstall the PHP-Java Bridge or continue trying to find a leak and fix it.
Any help would be appreciated.
Thank you.
Update 1
With MAT, I looked at the thread overview and stacks and you can see below that the PHP-Java Bridge contributes about 2/3 of the total heap memory of Tomcat. That's a lot of memory! I think there is a leak, I do. I can't find any information on the PHP-Java Bridge having inherit memory leak issues. But, to me, it appears that the problem is not that Tomcat is leaking. Ideas?
AppDynamics couldn't find any leaks, even when I manually added classes that were suspected in MAT. What I'm wondering is perhaps the PermGen error is a symptom of that case where the program has no leak and needs more PermGen memory allotted. It would be helpful to know if the PHP-Java Bridge is designed to eat a lot of memory, this much memory; maybe it's optimized for 64-bit, since the current setup is a 32-bit Java Web application. If I knew that this bridge needs a lot of memory, I would say OK, fine, and go from there. But it certainly appears as if there is a memory leak somewhere in the chain.
Update 2
I've been running Plumbr now for 2 hours and almost 10 minutes. I see that Tomcat memory is shooting up to 960 MB and probably will continue to climb. For those familiar with the program, the Java web application has been analyzed 3 times. So far, no leaks have been reported. If it stays this way, then the two conclusions I've arrived at are a) there are no leaks or b) there is a leak and, somehow, both AppDynamics and Plumbr missed it. If there are truly no leaks with this set of applications working together, then it must be that the Bridge uses a lot of memory and needs more PermGen memory than Tomcat's default, 64 MB -- at the very least, for 32-bit Java web applications.