Retrieving endpoints from kubernetes-etcd for haproxy

Question

I trying to retrieve kubernetes endpoints from my kubernetes-etcd-cluster for haproxy using confd.

When executing confd it always returns an error, that key '/registry' cant be found:

$ ./confd-0.12.0-linux-amd64 -config-file confd.toml
...
INFO Backend nodes set to https://10.31.9.49:2379
DEBUG Loading template resources from confdir /root/haproxy/kubernetes-endpoint-proxy/src/etc/confd/
DEBUG Found template: /root/haproxy/kubernetes-endpoint-proxy/src/etc/confd/conf.d/haproxy.toml
DEBUG Loading template resource from /root/haproxy/kubernetes-endpoint-proxy/src/etc/confd/conf.d/haproxy.toml
DEBUG Retrieving keys from store
DEBUG Key prefix set to /
ERROR 100: Key not found (/registry) [2540]

But listing in/below '/registry' with etcdctl works fine.

$ etcdctl \
--cacert=/etc/kubernetes/ssl/ca.pem \
--cert=./admin1.pem \
--key=./admin1-key.pem \
--endpoints 10.31.9.49:2379 \
get / --keys-only=true --prefix \
| grep endpoints
/registry/services/endpoints/default/echoheaders-x
/registry/services/endpoints/default/echoheaders-y
/registry/services/endpoints/default/kubernetes
/registry/services/endpoints/kube-system/default-http-backend
/registry/services/endpoints/kube-system/kube-controller-manager
/registry/services/endpoints/kube-system/kube-dns
/registry/services/endpoints/kube-system/kube-scheduler
/registry/services/endpoints/kube-system/kubernetes-dashboard

A single endpoints works as well:

etcdctl \
--cacert=/etc/kubernetes/ssl/ca.pem \
--cert=./admin1.pem \
--key=./admin1-key.pem \
--endpoints 10.31.9.49:2379 get /registry/services/endpoints/default/echoheaders-x
/registry/services/endpoints/default/echoheaders-x
k8s

v1  Endpoints�
�
echoheaders-xdefault"2/api/v1/namespaces/default/endpoints/echoheaders-x*$3bc4430d-ad97-11e7-8fe1-0022195f6b5b28B
                                                                                                                ������uZ
run
   echoheadersz�
z

10.244.1.66_
Poddefaultechoheaders-1076692255-rs1qm"$360a3956-ad97-11e7-8fe1-0022195f6b5b*22306754:worker05

�?TCP"

Listing the same endpoint as above with curl fails though:

curl \
--cacert /etc/kubernetes/ssl/ca.pem \
--cert ./master01.pem \
--key ./master01-key.pem  \
-L 'https://10.31.9.49:2379/v2/keys/registry/services/endpoints/default/echoheaders-x'
{"errorCode":100,"message":"Key not found","cause":"/registry","index":2540}

Why cant i list anything with curl, while etcdctl works?

Could anyone shed some light on this?

Answer 1

The answer to that is that etcd3 uses gRPC and provides access with API-Version 3. Confd and curl access the data using API-Version 2.

A gRPC <-> Json-Gateway is required: https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md

Answer comes froms https://github.com/coreos/etcd/issues/8682