Failure Err: Not able to connect to any etcd endpoints - etcd: 0/1 connected: kubeadm

Question

I tried to install cilium with coredns in kubeadm

kube: 1.12.3 cilium: 1.3.0

I get this error:

Readiness probe failed: KVStore:  Failure   Err: Not able to connect to any etcd endpoints - etcd: 0/1 connected: http://127.0.0.1:31079 - context deadline exceeded

I don't know why and if i need to install etcd on the master server.

kubectl get pods -n kube-system

cilium-9z4zd                                  0/1     Running   3          10m
cilium-s4x2g                                  0/1     Running   3          10m
coredns-576cbf47c7-44hp9                      1/1     Running   2          9m29s
coredns-576cbf47c7-6jst5                      1/1     Running   2          9m29s
etcd-ops-kube-master-dev                      1/1     Running   0          9m29s
kube-apiserver-ops-kube-master-dev            1/1     Running   0          9m29s
kube-controller-manager-ops-kube-master-dev   1/1     Running   0          9m26s
kube-proxy-79649                              1/1     Running   0          38m
kube-proxy-b56fk                              1/1     Running   0          38m
kube-scheduler-ops-kube-master-dev            1/1     Running   0          9m27s

Answer 1

I had a similar issue playing with Kubernetes the hard way, this was because of a wrong certificate

I did the following:

kubectl -n kube-system logs <etcd>

and found something like: embed: rejected connection from "172.17.0.3:36950" (error "remote error: tls: bad certificate", ServerName "")

I got the etcd config, you should have something like

$ kubectl -n kube-system get cm cilium-config -o yaml
apiVersion: v1
data:
  clean-cilium-bpf-state: "false"
  clean-cilium-state: "false"
  cluster-name: default
  ct-global-max-entries-other: "262144"
  ct-global-max-entries-tcp: "524288"
  debug: "false"
  disable-ipv4: "false"
  etcd-config: |-
    ---
    endpoints:
      - https://<ETCD_URL>:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    key-file: '/var/lib/etcd-secrets/etcd-client.key'
    cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
  legacy-host-allows-world: "false"
  monitor-aggregation-level: none
  sidecar-istio-proxy-image: cilium/istio_proxy
  tunnel: vxlan
kind: ConfigMap

Then I compared the keys of kubectl -n kube-system get secret cilium-etcd-client-tls -o yaml that provides 3 base64 values.

I can then test the keys using curl https://<ETCD_URL>:2379/v2/keys --cacert=etcd-client-ca.crt --cert=etcd-client.crt --key=etcd-client.key

You should then have something like {"action":"get","node":{"dir":true}}

Then, you can inspect the deployment, on my side, I have

kind: Deployment
metadata:
  labels:
    io.cilium/app: operator
    name: cilium-operator
  name: cilium-operator
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      io.cilium/app: operator
      name: cilium-operator
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.cilium/app: operator
        name: cilium-operator
    spec:
      containers:
      - args:
        - --kvstore=etcd
        - --kvstore-opt=etcd.config=/var/lib/etcd-config/etcd.config
        command:
        - cilium-operator
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CILIUM_DEBUG
          valueFrom:
            configMapKeyRef:
              key: debug
              name: cilium-config
              optional: true
        - name: CILIUM_CLUSTER_NAME
          valueFrom:
            configMapKeyRef:
              key: cluster-name
              name: cilium-config
              optional: true
        - name: CILIUM_CLUSTER_ID
          valueFrom:
            configMapKeyRef:
              key: cluster-id
              name: cilium-config
              optional: true
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: cilium-aws
              optional: true
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: cilium-aws
              optional: true
        - name: AWS_DEFAULT_REGION
          valueFrom:
            secretKeyRef:
              key: AWS_DEFAULT_REGION
              name: cilium-aws
              optional: true
        image: docker.io/cilium/operator:latest
        imagePullPolicy: Always
        name: cilium-operator
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/etcd-config
          name: etcd-config-path
          readOnly: true
        - mountPath: /var/lib/etcd-secrets
          name: etcd-secrets
          readOnly: true
      dnsPolicy: ClusterFirst
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: cilium-operator
      serviceAccountName: cilium-operator
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: etcd-config
            path: etcd.config
          name: cilium-config
        name: etcd-config-path
      - name: etcd-secrets
        secret:
          defaultMode: 420
          optional: true
          secretName: cilium-etcd-secrets```