"Lag in MB" patroni/postgresql how do i troubleshoot it?

Question

The replica's is getting behind. Where do i begin to troubleshoot this? I have looked in /var/log/patroni/patroni.log but don't see anything weird.

                                        Lag in MB

postgres2 | 192.168.1.26 | Replica | running | 5 | 5298

postgres3 | 192.168.1.27 | Replica | running | 5 | 5298

Answer 1

With 5 GB of lag, you're often left to reinitialize the affected Patroni node (i.e., delete contents of data and tablespace directories, then do a patronictl reinit). A reason why this could have happened is that you ran out of space on your disk/tablespace. That is, when no further DB updates can be written to disk, lag grows indefinitely.

So, the lag you've witnessed is just a symptom of a multitude of things that could have gone wrong. Checking the Patroni log, as you have done, is good, but check also the PostgreSQL log. Fortunately, reinitialization of a node is usually not a big deal.