Optimizing Flink's memory usage

Question

I meet a scenario to deploy Flink on some IoT devices, so what I concern about is the limited resources on these platforms, the default configuration results in unacceptable memory usage to me. I'm trying to modify the memory configuration(of taskmanager) in flink-conf.yaml according to official docs and some blogs. Here are what I found and was confused:

• If I set the Task Heap to 0 the jobs can still run normally, and results through operators still remain correct. Are Task Heap and Framework Heap shared so the tasks can utilize heap memory other than Task Heap?

• I also set the Managed Memory to 0 since I'm not using RocksDB as state backend, so the intermediate states should be stored on the heap memory. I found an interesting thing here: the size of Task Heap configured in flink-conf.yaml is related to the runtime memory usage of taskmanager. For example, by default it's set to 256MB, when a job is running the memory usage of taskmanager can be up to 800MB. When I set it to a small number instead, like 10MB, memory usage of taskmanager would be at most 500MB. And from the Flink UI I noticed that JVM would run GC much more frequently when Task Heap is small. I can't find corresponding explain from Flink's docs. I guess there is a space to store states somewhere on the heap, the space size is related to Task Heap, and every time it's fullfilled the JVM will call GC.

• About the metrics shown on Flink UI, the part of Network is reflected in real memory occupation even though it's "not actually used". For example, if Network is set to 100MB, taskmanager will take 90MB more memory compared to when it's set to 10MB, but most of time the usage of Network is less than 1MB.

• I'm also wondering whether the memory occupied by taskmanager, as a JVM process, is exactly or approximatively equal to Heap + Non-Heap + Outside JVM Memory? When there is no job running I think it's reasonable, but after a job is submitted, taskmanager will occupy a few hundred MBs more memory but I can's find corresponding terms on Flink UI.

Many thanks if you can explain my problems above.