We are an ActiveMQ
setup with all default settings. Its broker url is like failover:(tcp://host1,tcp://host2,tcp://host3)?randomize=false
We have 2 message Queues where producer publishes persistent messages.
Messages generally have a steady flow but when markets open then there is burst of messages for almost 2 hours. This setup works fine for 2-3 days and then consumer slows down in processing messages and we see queue build ups.
Consumer performs following steps:
- Transform message.
- Process it
- Add to cache and DB.
Prefetch size
is 1000.
Following are some of the observations:
- Consumer processing doesn't seem to be the issue else it would have always slowed down.
- Looks like ActiveMQ broker slows down in processing but i dont have evidence
- We have observed that out of 3 hosts only host 1 and 2 process majority of messages.
Any ideas on how to explain this slowness?
EDIT:
Analysed this further and found that Kahadb
store [Persistent
messages are stored in Kahadb
in broker] reaches 100% storage when consumer slows down.
I can understand that producer is blocked when storage reaches 100% but why would consumer slow down? Is it because Kahadb
may slow down during this time and extracting messages equivalent to prefetch size takes more?