Activemq high memory consumption

Question

I am using ActiveMQ 5.8.0 along with puppet mcollective 2.5.2, Java 1.8.0_31

And I am facing memory issue in AciveMQ. The problem is ActiveMQ is consuming memory and it is not getting free after that. It keeps on increasing until we restart ActiveMQ. I don't know what is the issue with this, might be some configuration missing or garbage collection settings. Below are the config files.

activemq.xml

<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="locations">
        <value>file:${activemq.base}/conf/credentials.properties</value>
    </property>
</bean>

<!--
    The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" deleteAllMessagesOnStartup="true">

    <!--
      For better performances use VM cursor and small memory limit.
      For more information, see:

      http://activemq.apache.org/message-cursors.html

      Also, if your producer is "hanging", it's probably due to producer
      flow control.

      For more information, see:
      http://activemq.apache.org/producer-flow-control.html
    -->

    <destinationPolicy>
        <policyMap>
          <policyEntries>
            <policyEntry topic=">" producerFlowControl="true" memoryLimit="100mb">
              <pendingSubscriberPolicy>
                <vmCursor />
              </pendingSubscriberPolicy>
            </policyEntry>
            <policyEntry queue=">" producerFlowControl="true" memoryLimit="100mb" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" >
              <!--
                Use VM cursor for better latency
                For more information, see:

                http://activemq.apache.org/message-cursors.html

              <pendingQueuePolicy>
                <vmQueueCursor/>
              </pendingQueuePolicy>
              -->
            </policyEntry>
          </policyEntries>
        </policyMap>
    </destinationPolicy>

    <!--
      The managementContext is used to configure how ActiveMQ is exposed in
      JMX. By default, ActiveMQ uses the MBean server that is started by
      the JVM. For more information, see:

      http://activemq.apache.org/jmx.html
    -->
    <managementContext>
        <managementContext createConnector="false"/>
    </managementContext>

    <!--
      Configure message persistence for the broker. The default persistence
      mechanism is the KahaDB store (identified by the kahaDB tag).
      For more information, see:

      http://activemq.apache.org/persistence.html
    -->
    <persistenceAdapter>
        <kahaDB directory="${activemq.base}/data/kahadb"/>
    </persistenceAdapter>

    <plugins>
      <!--
        Enable the statisticsBrokerPlugin to allow ActiveMQ to collect
        statistics.
      -->
      <statisticsBrokerPlugin/>

      <!--
        Here we define a default set of users
      -->
      <simpleAuthenticationPlugin>
        <users>
          <authenticationUser username="mcollective" password="secret" groups="mcollective,admins,everyone"/>
        </users>
      </simpleAuthenticationPlugin>
      <authorizationPlugin>
        <map>
          <authorizationMap>
            <authorizationEntries>
              <authorizationEntry queue=">" write="mcollective" read="mcollective" admin="mcollective" />
              <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
              <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
              <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
              <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
            </authorizationEntries>
          </authorizationMap>
        </map>
      </authorizationPlugin>
    </plugins>


    <!--
      The systemUsage controls the maximum amount of space the broker will
      use before slowing down producers. For more information, see:

      http://activemq.apache.org/producer-flow-control.html
    -->
    <systemUsage>
        <systemUsage>
            <memoryUsage>
                <memoryUsage limit="20 mb"/>
            </memoryUsage>
            <storeUsage>
                <storeUsage limit="1 gb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="100 mb"/>
            </tempUsage>
        </systemUsage>
    </systemUsage>

    <!--
      The transport connectors expose ActiveMQ over a given protocol to
      clients and other brokers. For more information, see:

      http://activemq.apache.org/configuring-transports.html
    -->
    <transportConnectors>
        <transportConnector name="openwire" uri="tcp://0.0.0.0:6166"/>
        <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
    </transportConnectors>

</broker>

<!--
  Enable web consoles, REST and Ajax APIs and demos
  It also includes Camel (with its web console), see ${ACTIVEMQ_HOME}/conf/camel.xml for more info

  Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/>

activemq-wrapper.conf

wrapper.debug=FALSE
set.default.ACTIVEMQ_HOME=/usr/share/activemq
set.default.ACTIVEMQ_BASE=/usr/share/activemq
set.default.ACTIVEMQ_CONF=/usr/share/activemq/conf
wrapper.working.dir=/var/log/activemq

# Java Application
wrapper.java.command=java

# Java Main class.  This class must implement the WrapperListener interface
#  or guarantee that the WrapperManager class is initialized.  Helper
#  classes are provided to do this for you.  See the Integration section
#  of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp

# Java Classpath (include wrapper.jar)  Add class path elements as
#  needed starting from 1
wrapper.java.classpath.1=/usr/share/java/tanukiwrapper.jar
wrapper.java.classpath.2=%ACTIVEMQ_HOME%/bin/activemq.jar

# Uncomment to add mysql support
#wrapper.java.classpath.3=/usr/share/java/mysql-connector-java.jar

# Java Library Path (location of Wrapper.DLL or libwrapper.so)
# Handle both 32bit and 64bit tanukiwrapper
wrapper.java.library.path.1=/usr/lib
wrapper.java.library.path.2=/usr/lib64

# Java Additional Parameters
# note that n is the parameter number starting from 1.
wrapper.java.additional.1=-Dactivemq.home=%ACTIVEMQ_HOME%
wrapper.java.additional.2=-Dactivemq.base=%ACTIVEMQ_BASE%
wrapper.java.additional.3=-Dcom.sun.management.jmxremote
wrapper.java.additional.4=-Dorg.apache.activemq.UseDedicatedTaskRunner=true

# Example configuration for SSL
#wrapper.java.additional.n=-Djavax.net.ssl.keyStorePassword=password
#wrapper.java.additional.n=-Djavax.net.ssl.trustStorePassword=password
#wrapper.java.additional.n=-Djavax.net.ssl.keyStore=%ACTIVEMQ_BASE%/conf/broker.ks
#wrapper.java.additional.n=-Djavax.net.ssl.trustStore=%ACTIVEMQ_BASE%/conf/broker.ts

#wrapper.java.additional.n=-Dderby.storage.fileSyncTransactionLog=true

# Uncomment to enable jmx
#wrapper.java.additional.n=-Dcom.sun.management.jmxremote.port=1616
#wrapper.java.additional.n=-Dcom.sun.management.jmxremote.authenticate=false
#wrapper.java.additional.n=-Dcom.sun.management.jmxremote.ssl=false

# Uncomment to enable YourKit profiling
wrapper.java.additional.n=-Xrunyjpagent

# Uncomment to enable remote debugging
#wrapper.java.additional.n=-Xdebug -Xnoagent -Djava.compiler=NONE
#wrapper.java.additional.n=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3

# Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=512
wrapper.java.maxmemory=4096

# Application parameters. activemq.xml will be loaded from the config directory
wrapper.app.parameter.1=org.apache.activemq.console.Main
wrapper.app.parameter.2=start
wrapper.app.parameter.3-xbean:activemq.xml

#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM

# Log Level for console output.  (See docs for log levels)
wrapper.console.loglevel=INFO

# Log file to use for wrapper output logging.
wrapper.logfile=%ACTIVEMQ_BASE%/log/wrapper.log

# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM

# Log Level for log file output.  (See docs for log levels)
wrapper.logfile.loglevel=INFO

# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=10m

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=7

# Log Level for sys/event log output.  (See docs for log levels)
wrapper.syslog.loglevel=NONE

#********************************************************************
# Wrapper Windows Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=ActiveMQ

#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Name of the service
wrapper.ntservice.name=ActiveMQ

# Display name of the service
wrapper.ntservice.displayname=ActiveMQ

# Description of the service
wrapper.ntservice.description=ActiveMQ Broker

# Service dependencies.  Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=

# Mode in which the service is installed.  AUTO_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START

# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false

Anybody please help. Thanks.

Answer 1

This is nothing unique about ActiveMQ - but pretty much how Java applications can behave. However, it will stay below the max memory limit, 4096 in your config. Nothing is actually wrong. Over time, the JVM may return some memory to the system if the java process (ActiveMQ in this case) stays low for a long period of time. The exact mechanics may differ between different JVM implementations and settings. This article explains a bit, but is a bit outdated.

Refer to this thread for some more discussion around JVM and memory management. java.exe process uses more memory and does not free it up

If you want to visualize the actual memory usage (not the memory allocated to the java process), then I suggest you connect to the ActiveMQ JVM using jvisualvm. It's usually bundled with the JDK.