I've been experimenting with Flink streaming for a while, using benchmarks like the Yahoo streaming benchmark: https://github.com/yahoo/streaming-benchmarks which are supposed to stress the system but I never achieved a satisfying CPU utilization - in fact it was mainly as low as ~25% using all available system cores (parallelism = nodes*cores) and one TaskManager slot per core.
Recently, I started working with Gelly, Flink's Graph API using some of the provided example algorithms (e.g. Pagerank), batch-processing datasets varying from tens of thousands to hundreds of millions vertices.
I occupy four TaskManagers of 32 cores each, and as suggested by the documentation I set taskmanager.numberOfTaskSlots: 32 and parallelism.default: 128.
Even if I increase these values, the average CPU utilization never reaches above 40%. Consequently, I achieve low performance as my resources are not fully utilized.
I also want to point out the fact that in some cases I have noticed better performance with lower parallelism levels (and CPU utilization).
What am I missing?
The more data in the source with more unique keys and more aggregation operators must bring your CPU levels high. For example, I have 250k messages into kafka per second (flink source reads as much per sec), with almost 170K unique keys, 5 min tumbling window and 15 aggregations per message. I have a 3 node, 16 core flink cluster, with source parallelism 15, aggregation parallelism 48 and sink parallelism 3. My CPU usage is constantly around 85-90% in each node.
You could just pump more messages in, and duplicate your aggregation logic to perform more operations on each message. Try to get more unique keys out of every message.