mariadb galera node stuck at WSREP state transfer ongoing

Question

I restarted mariadb galera cluster node, and now its taking ages to start, however I noticed Status: "WSREP state transfer ongoing, current seqno: 9331 waited 510.000000 secs" and also noticed Memory: 71.3M memory usage is going up everytime I recheck mariadb service status

 mariadb.service - MariaDB 10.1.47 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fri 2020-11-27 21:26:43 GMT; 8min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 19409 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, statu
  Process: 19407 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 19497 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /u
  Process: 19495 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status
  Process: 19494 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=
 Main PID: 19701 (mysqld)
   Status: "WSREP state transfer ongoing, current seqno: 9331 waited 510.000000 secs"
    Tasks: 14 (limit: 4573)
   Memory: 71.3M
      CPU: 49.181s
   CGroup: /system.slice/mariadb.service
           ├─19701 /usr/sbin/mysqld --wsrep_start_position=5b96f94b-2dcb-11eb-8e8b-eff8238871c4:9331
           ├─19764 sh -c wsrep_sst_rsync --role 'joiner' --address '192.168.5.165' --datadir '/var/lib/mysql/' 
           ├─19765 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address 192.168.5.165 --datadir /var
           ├─19824 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           ├─19865 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           ├─19884 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf
           └─23405 sleep 1

what should I do, what is the best possbile way to avoid this in the future ?

Answer 1

First of all support for MariaDB 10.1 has ended but you could increase the systemd timeout.