I am running a JBoss 5.1.0.GA clustered setup with 4 nodes. This makes use of JBoss Messaging 1.4.3 to receive messages from other applications and process within deployed applications in these JBoss instances. I've run into an issue where the JBoss messaging appears to become deadlocked and JBoss stops processesing messages despite them existing in the queue. There are no clear errors in the logs as to why this happens. This happens over some time (~1 week) and is not easily reproducible in any sort of testing environment.
This is running on CentOS5, using Java 6.
I pulled stack trace from each node to try to get to the root cause, but I have not been able to garner anything useful from these.
I have a made a few recent changes to the applications deployed, but no application code appears in any of the threads in the thread dumps. One other recent change made that may have an impact was deploying a newer netty jar, netty-3.6.3.Final.jar to /common/lib.
I have included the thread dumps here: Thread Dump
Any suggestions are greatly appreciated!