How to recovery cluster when all nodes down?

Question

If all nodes in a 3-nodes Percona Cluster have shutdown (gracefully shutdown or crash), from this blog, it says that when the nodes could reach each other, the cluster could recover automatically. However, starting the nodes in such a situation seems a difficult task.

So is there a reliable and operable method to do cluster recovery in this situation?

Answer 1

Examine the grastate.dat file on all 3 nodes. Which node has the highest sequence number? You should bootstrap that node. Wait for it to come online. Then start node2. It should IST from the bootstrap node. Then start node3.

Golden rule: You must always bootstrap the very first node of any cluster. Bootstrapping does not erase data; it only starts a new cluster.

Depending on the version, you may need to set safe_to_bootstrap in the grastate file to 1 manually.

Answer 2

Another thing you can try to check which is most advanced node run below command on every node and check which node has largest committed transaction value.

mysqld_safe --wsrep-recover

start First node which has max committed value then second and third