Confd error: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

Question

While debugging I realised that confd doesn't pick up the keys and my journal looks like this:

Sep 18 18:31:50 ip-10-171-54-76.ec2.internal docker[24891]: [nginx] waiting for confd to refresh nginx.conf Sep 18 18:31:56 ip-10-171-54-76.ec2.internal docker[24891]: 2014-09-18T18:31:56Z 9122c7a54edc confd[9572]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

I use nsenter to log in to the running container to run some experiments for debugging purposes. I ran this command

confd -onetime -node 172.17.42.1:4001 -config-file /etc/confd/conf.d/nginx.toml

Then received this error as above

confd[12894]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

I am totally clueless at this point. I am using EC2 with the stable version of CoreOS and I am sure that etcd is running on the host. Also, I can ping the host from inside the container successfully.

Any ideas on what's wrong?

Assistance will be much appreciated.

Answer 1

This error indicates that your etcd cluster isn't operating correctly, so confd has nothing to watch. It has probably lost quorum. The logs (journalctl -u etcd) should indicate what happened.