journalctl unit logging stopped while unit is still running

Question

Can't seem to find anything on it and is a weird one too me.

I have a python program that inserts database records very frequently, and its still doing that, so I know the program is still running.

I have the program set as a service unit in systemctl with logging to journalctl.

Logs have been great for a few months, but at 1 oclock today logs stopped being added to the log data for this unit.

Is this just a memory problem? I don't know why but I thought journalctl would handle that.

Answer 1

Yes it was a memory problem, solved by finding where journalctl stores its logs by running

man systemd-journald

I then wanted to check the remaining memory on my system so I ran:

df

This told me that the top level directory of my logs storage directory was 100% full. So no more room to store.

I think I have an old version of journalctl and do not have the --rotate flag, meaning it was difficult to remove non-archived logs. I was able to free up 10% of space with:

journalctl --vacuum-time=1d

I then ran:

sudo systemctl restart systemd-journald

This restarts the journal service, this didn't get my logs working though so I then restarted my own service with:

systemctl restart service.name

Then my logs started showing again!