Getting ETCD Metrics from etcdctl

Question

I am trying to get ETCD metrics like the number of ETCD keys and size as well as the number of requests made to ETCD through exec (ing) into a kubernetes pod (etcdctl) and am not sure what command to use for this.

An alternative (such as cUrl) would help as well.

Thanks for the help!

Answer 1

You need to extract the information from etcd and filter what you want. To illustrate, I will show you how to get the number of total keys from etcd.

NOTE: Tested in kubernetes 1.18.2.

# Getting etcd pod IP and set a local variable:
ADVERTISE_URL="https://$(kubectl get pods -n kube-system -l=component=etcd -o=jsonpath='{ .items[*].status.podIP }'):2379"

# Getting ectd pod name and set a variable ETCD_POD
ETCD_POD=$(kubectl get pods -n kube-system -l=component=etcd -o=jsonpath='{ .items[*].metadata.name}')

# Extracting all etcd keys/values to a file called "etcd-kv.json":
kubectl exec $ETCD_POD -n kube-system -- sh -c \
"ETCDCTL_API=3 etcdctl \
--endpoints $ADVERTISE_URL \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--key /etc/kubernetes/pki/etcd/server.key \
--cert /etc/kubernetes/pki/etcd/server.crt \
get \"\" --prefix=true -w json" > etcd-kv.json

Now you have all the keys/values pairs from etcd, you just need to filter to extract the information you need. For example, to list all keys you can use the command:

for k in $(cat etcd-kv.json | jq '.kvs[].key' | cut -d '"' -f2);  do echo $k | base64 --decode;  echo;  done

and to count the number of keys, just use the command wc -l on the end of this command, like:

for k in $(cat etcd-kv.json | jq '.kvs[].key' | cut -d '"' -f2);  do echo $k | base64 --decode; echo; done | echo "Total keys=$(wc -l)"
Total keys=308

References:

A closer look at etcd: The brain of a kubernetes cluster