How to handling Database Issues in a GenServer and the Impact of Max Restarts in Elixir Supervisor

Question

I have an application that displays messages on a website. These messages are shown after performing some calculations using my service. My main question is related to the behavior inside the GenServer. In the GenServer, I make a query to retrieve messages with certain conditions from the database. However, if there is a problem with the database connection or if the database goes down, the message query function may raise an exception.

To handle such situations, I have set the max_restart option to a very high value (e.g., 100000000) in the supervisor configuration. This means that even if the GenServer restarts multiple times due to errors, the supervisor will continue restarting it. This approach helps me handle database connection issues or outages.

However, I would like to know the impact of setting such a high value for max_restart in the supervisor on system performance. Does it affect the system's overall performance? Additionally, I'm curious to understand what happens to the previous state's data when the supervisor restarts the GenServer. Furthermore, I have limited knowledge about the inner workings of the Elixir garbage collection process and the Erlang virtual machine (VM).

Please help me clarify these questions as I do not have much experience in low-level system details or extensive knowledge about the Erlang VM.

Answer 1

The main problem with setting max_restart high is that you could potentially exacerbate the conditions that are impacting the database availability by repeatedly attempting to reconnect in rapid succession without backing off. It depends on the nature of the failure though; a max_restart that takes 5-10 seconds for each restart attempt (as in a timeout) may not hurt that much. So you'll probably need to observe what a typical failure looks like before you can answer that.

If you rapidly restart a process that dies quickly, there may be some overall system performance impact, but it's hard to predict that impact without triggering and measuring failure conditions.

Keep in mind there are other alternatives to max_restart. You could, for example, have another process observe a supervisor with Process.monitor/1, and implement def handle_info({:DOWN, ref, _, _pid, _reason}, state) in the observer process, then, for example, schedule a restart after a backoff period.

If you're implementing the observing process as a GenServer, I believe you will need to call Process.flag(:trap_exit, true) so that the GenServer process doesn't do its default behavior of dying when it receives an EXIT or DOWN message from a linked process.