Analyzed memory dump shows only <Unresolved> references with DotMemory

Question

I have a memory leak problem on a AWS dockerized microservice deployed with ECS. I wanted to analyze the dump with DotMemory so I used exec to connect to the container, gcore to save the dump and then transfer that dump to an S3 bucket so I could download it. Problem is, when I open the dump everything is "Unresolved" and I can't understand what's going on.

I've tried to run the same microservice locally and take a memory dump with Windows Task Manager and everything worked fine. Sadly, since is a complex system I can't replicate locally exactly what's happening when it's deployed so I need to create a readable dump from my deployed microservice. How do I fix the issue?

Answer 1

This issue might appear if the dump is missing the segments with metadata.

Since kernel 2.6.23, the Linux-specific /proc/[pid]/coredump_filter file can be used to control which memory segments are written to the core dump file in the event that a core dump is performed for the process with the corresponding process ID.

See core man section "Controlling which mappings are written to the core dump" for more information.

In order to get a proper dump of the dotNet process coredump_filter should be set to at least 0x3f.

You can check what the current filter set for your process by executing:

cat /proc/<pid>/coredump_filter

To set the proper coredump_filter type:

echo "0x3f" > /proc/<pid>/coredump_filter

<pid> should be replaced with your process ID, for example:

echo "0x3f" > /proc/144/coredump_filter

UPDATE: May 10, 2023

I've managed to collect a full dump with all the necessary information by using createdump utility:

createdump -u -f /tmp/coredump <pid>

The -u option tells createdump to generate a full memory dump, including all memory-mapped files. Note that this will result in a very large dump file for applications that use a lot of memory or have many memory-mapped files.

Example:

docker container exec -it \
  -u root --privileged \
  <container-id> \
  /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.16/createdump -u -f /tmp/coredump <pid>

docker cp <container-id>:/tmp/coredump .