Tried googling for this but couldn't find any answers. I was profiling my golang app and pprof showed it running around ~200 mb for inuse_space but then I checked it in Prometheus and it showed it using over 900mb of memory. I'm inclined to trust Prometheus since it's pulling from the node itself but I'm trying to understand how pprof isn't recognizing 700 mb worth of memory.
As an example: This is pprof output
Type: inuse_space
Time: Apr 11, 2022 at 1:12pm (CDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) granularity=lines
(pprof) top50
Showing nodes accounting for 176.69MB, 96.64% of 182.82MB total
Dropped 41 nodes (cum <= 0.91MB)
Showing top 50 nodes out of 70
meanwhile Prometheus shows this for memory usage.
EDIT:
For reference, here is the Prometheus query:
max(container_memory_working_set_bytes{container="$container",pod="$pod"} / (1024*1024) ) by (container)
Why 'Total MB' in golang heap profile is less than 'RES' in top?
^ Seems to be the answer. The app is generating ~800mb of space that the GC is holding onto in case the app needs it but its only using ~1/4th of it at any given time which explains the difference.