ActiveMQ consumer slows down after few days

727 Views Asked by At

We are an ActiveMQ setup with all default settings. Its broker url is like failover:(tcp://host1,tcp://host2,tcp://host3)?randomize=false

We have 2 message Queues where producer publishes persistent messages.

Messages generally have a steady flow but when markets open then there is burst of messages for almost 2 hours. This setup works fine for 2-3 days and then consumer slows down in processing messages and we see queue build ups.

Consumer performs following steps:

  1. Transform message.
  2. Process it
  3. Add to cache and DB.

Prefetch size is 1000.

Following are some of the observations:

  1. Consumer processing doesn't seem to be the issue else it would have always slowed down.
  2. Looks like ActiveMQ broker slows down in processing but i dont have evidence
  3. We have observed that out of 3 hosts only host 1 and 2 process majority of messages.

Any ideas on how to explain this slowness?

EDIT:

Analysed this further and found that Kahadb store [Persistent messages are stored in Kahadb in broker] reaches 100% storage when consumer slows down. I can understand that producer is blocked when storage reaches 100% but why would consumer slow down? Is it because Kahadb may slow down during this time and extracting messages equivalent to prefetch size takes more?

0

There are 0 best solutions below