The servers are alive but the clients aren't! "No installed keys could decrypt the message", how to resolve this on consul?

Question

Once I enabled encrypt, all my consul cluster failed. this is sudo systemctl consul status:

memberlist: failed to receive: No installed keys could decrypt

Nov 01 08:49:30 server-1 consul[593217]: 2021-11-01T08:49:30.031+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37442
Nov 01 08:49:58 server-1 consul[593217]: 2021-11-01T08:49:58.992+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41492
Nov 01 08:49:59 server-1 consul[593217]: 2021-11-01T08:49:59.882+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35558
Nov 01 08:50:00 server-1 consul[593217]: 2021-11-01T08:50:00.042+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37460
Nov 01 08:50:29 server-1 consul[593217]: 2021-11-01T08:50:29.004+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41510
Nov 01 08:50:29 server-1 consul[593217]: 2021-11-01T08:50:29.895+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35576
Nov 01 08:50:30 server-1 consul[593217]: 2021-11-01T08:50:30.056+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37478
Nov 01 08:50:59 server-1 consul[593217]: 2021-11-01T08:50:59.018+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.4:41528
Nov 01 08:50:59 server-1 consul[593217]: 2021-11-01T08:50:59.909+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.12:35594
Nov 01 08:51:00 server-1 consul[593217]: 2021-11-01T08:51:00.067+0330 [ERROR] agent.server.memberlist.lan: memberlist: failed to receive: No installed keys could decrypt the message from=10.10.10.7:37496

The ACL and TLS are commented out, I even comment encrypt in all of my clients, but still received the above output. How am I able to resolve this? Thanks.

Answer 1

You must have exactly the same encrypt settings on both types of node: server and client. Or, you must have encryption disabled everywhere. You can consult the official tutorial of step-by-step enabling encryption on the existing cluster.

UPD: Some useful commands to use one key in all nodes (both client and server).

Get keys info from all nodes

consul keyring -list

Generate new key

consul keygen

Broadcast new key to all nodes

consul keyring -install=<<KEY>>

Switch to a new key

consul keyring -use=<<KEY>>