How to recover ActiveMQ from full KahaDb persistence store?

Question

Using AMQ 5.9 I had a network issue that caused messages in my queue to back up. I started getting log messages

"Usage (default:store:queue://myqueue:store) percentUsage=101% usage=6254005794, limit=6144430090, ...Stopping producer...to prevent flooding queue..)

I have not been able to recover and start consuming messages again.

I tried going into activemq.xml to increase my max store usage:

 <systemUsage>
    <systemUsage>
         <storeUsage>
             <storeUsage limit="15 gb" />
         </storeUsage>
    </systemUsage>
  </systemUsage>

I also tried to turn off flow control with

  <policyEntry queue=">" producerFlowControl="false"/>

But I still get the same error message.

I have the disk space. There are no settings being overridden on the command line. How can I recover and get my messages processed?

Answer 1

I ended up finding an ugly (yet effective) way to get around this. If you connect to activemq via JMX (jconsole), you can navigate to the mbean org.apache.activemq.Broker and find attribute StoreLimit. You can MANUALLY increase this value in JConsole then activemq will restart message processing shortly thereafter.