I have been exploring Apache Storm for one of my use case. Spout will read the data from my own kafka implemenation and it will pass to bolts.
My topology is working faster for single consumer group. If I increased the number of consumer groups inside the topology, I could see processing slowness though I have spout and bolts dedicated to each partition of all the consumer groups.
When I debug, I found that there are more number of context switches happened during my processing and therefore logged the thread name processing the bolts. I could see, some times different executor thread is passing the same bolt id.
I am suspecting due to this there may be context switch may happen and due to cpu thread cache miss, there may be increase in the latency.
Whether my assumption is correct. If this is the behaviour, is there any option to set the task by same executor thread instead of passing it to the other executor thread.
After further analysis and debugging, I found that there is CPU steal in the vm in which the storm processes is running. Hence I thought about whether the number of cores would be a problem.
Hence I ran the same process in the machine with a high number of core and cpu steal percentage has been reduced and also I didn't any see slowness with my processing. So all these problems are with machine configuration it seems.