Strange phenomenon in Open MQ: unexpectedly messages start to get stuck in the JMS Queue for a while before delivering them to the MDB

Question

Brief overview of the application

The JEE application runs in GlassFish server, see the system environment details below and uses the bundled Open MQ configured in embedded mode in GlassFish for internal JMS messaging. As message consumer Message Driven Bean is used. The MDB pool size is the default 32, see the JMS configuration and MDB class details below.

The application can be divided into two main parts and functions roughly as follows:

• the external system adapter listens to legacy messages from various external systems and sends them as JMS messages to the core application.
• the core application processes the received JMS messages. The MDB runs here.

Problem description

The message delivery from the adapter to the core works usually smoothly taking only couple of seconds. However due to an unknown reason sometimes the following situation happens permanently: the message sent by the adapter get stuck in the JMS queue for a minute and only thereafter it is passed to the MDB. Using the Open MQ monitoring commands: imqcmd metrics dst -t q -n CacheQueueDest or imqcmd query dst -t q -n CacheQueueDest we can see the sent message is in the queue and only in a minute the onMessage method of the MDB gets invoked. When this faulty situation happens even one message will get stuck in the queue so the problem will exist independent of the system load.

I have set DEBUGHIGH log level for Open MQ but have found nothing in the log. I have configured Open MQ as local (running in own process instead of within GlassFish process) and set packet level logging but likewise have found nothing.

The only way to get rid of this faulty state is to restart the GlassFish server.

I would appreciate any ideas to solve this issue.

System environment:

• SunOS 5.10 Generic_150401-05 i86pc i386 i86pc

• GlassFish Server Open Source Edition 3.1.2.2 (build 5)

• Open Message Queue 4.5.2
  Oracle
  Version: 4.5.2 Patch 1 (Build 3-d)
  Compile: Thu Jun 7 10:46:15 PDT 2012
  

• java version "1.7.0_25"
  Java(TM) SE Runtime Environment (build 1.7.0_25-b34)
  Java HotSpot(TM) 64-Bit Server VM (build 23.25-b03, mixed mode)
  

JMS configuration from domain.xml:

<resources>
    <admin-object-resource 
     res-adapter="jmsra"
     res-type="javax.jms.Queue"
     jndi-name="jms/cacheQueue">
        <property name="Name" value="CacheQueueDest"></property>
    </admin-object-resource>

    <connector-connection-pool 
         name="jms/cacheConnFactory" 
         resource-adapter-name="jmsra" 
         is-connection-validation-required="true"
         connection-definition-name="javax.jms.QueueConnectionFactory"
         max-wait-time-in-millis="6000"
         fail-all-connections="true"
         transaction-support="NoTransaction"/>
</resources>

<configs>
    <config name="server-config">

        <mdb-container/>

        <jms-service default-jms-host="default_JMS_host">
            <jms-host port="59900" host="localhost" name="default_JMS_host"></jms-host>
        </jms-service>
    </config>
</configs>

MDB class:

...
import javax.jms.Message;
import javax.jms.MessageListener;
...
@MessageDriven(mappedName = "jms/cacheQueue", messageListenerInterface = javax.jms.MessageListener.class)
public class CacheChange implements MessageListener {
    ...
    @TransactionAttribute(value = TransactionAttributeType.NOT_SUPPORTED)
    @Override
    public void onMessage(final Message message) {
        ...
    }
    ...
}

Answer 1

I found this problem in sometime in embbed mode , We switch to local standalone mode( start OpenMQ separate JVM ), after that the problem has been solved. you can try.

Tomz.