How to monitor a systemd service via http endpoint

Question

I have a process I would like to monitor consul-template which is a process currently managed by systemd. I use Pandora to monitor an HTTP endpoint for my other processes, however consul-template doesn't have an HTTP endpoint.

I've thought of a couple of approaches:

One is to launch a HTTP server with python that will run when the consul-template process is run, the problem is I am not sure I can guarantee that the HTTP server will exit when the process exits. Also I can't guarantee that the HTTP server as simple as it is wont crash independently.

The other solution is to install Monit which is a fully featured monitoring service and just use that instead of systemd to do process management too. This approach will mean I have to monitor Pandora AND Monit now and set up alerts accordingly, I would much rather just get an HTTP endpoint up so I can monitor from Pandora only.

Answer 1

systemd doesn't have a native HTTP server, but the system can queried over a network via SSH. You'll find a --host option for systemctl. For example, you could get the status over the network in a machine-readable format like this:

 systemctl --host [email protected] status consol-template

Look for return values like:

 ActiveState=active
 SubState=running

To confirm that the service is up and running.

You could either use a cron job to push or pull this status data regularly to a place that Pandora can access it, or see if there's a way for Pandora to check the output of a command run over SSH.

If the machine hosting Pandora doesn't have systemctl installed, you could still use the same general approach to remotely execute the status command over ssh:

 ssh [email protected] systemctl status consol-template