I have a flink job with overall parallelism of 4. There are two component which are memory and CPU intensive, for which I have created slotsharing group (H). Currently I have 2 task manager with 2CPU core each having 4 slot. When i deploy my job most of time the slotsharing components are not evenly distributed among task manager, which means at times 1 task manager have 4 instance of slotSharing group(H), ideal would be 2 on each task manager so that memory and cpu allocated to each task manager can be used optimally by the slotsharing group (H).
I started looking into https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/finegrained_resource/
Two question
- If I specify 0.75 Cu for SlotSharingGroup (H) and 0.25 for default slotSharingGroup (D), will it force to enable only 2 instance of operators from H on each machine ?
- How will GC behave ? say in this case if TM has 4GB and I allocate 3GB to H and 1GB to D, if H reaches close to 3GB but D is still free, will ZGC starts cleaning up unused memory for H ? this is where main confusion is since ZGC is at JVM level, how can ZGC know that memory allocated to a particular slot is about to be fill ?