I have an application running in Docker that leaks memory over time. I need to restart this application periodically when memory usage gets over a threshold. My application can respond to signals or by touching tmp/restart.txt (this is Rails)... as long as I can run a script or send a configurable signal when limits are triggered, I can safely shut down/restart my process.
I have looked into limiting memory utilization with Docker, but I am not seeing a custom action when a limit or reservation is hit. A SIGKILL would not be appropriate for my app... I need some time to clean up.
I am using runit as a minimal init system inside the container, and ECS for container orchestration. This feels like a problem that is attended to at the application or init level... killing a container rather than restarting the process seems heavy.
I have used Monit for this in the past, but I don't like how Monit deals with pidfiles... too often Monit loses control of a process. I am trying out Inspeqtor which seems to fit the bill very well, but while it supports runit there are no packages that work with runit out of the box.
So my question is, if SIGKILL is inappropriate for my use case, what's the best way to monitor a process for memory usage and then perform a cleanup/restart action based on that usage crossing a threshold?