Metrics server not working in Kubernetes cluster

Question

I have set up kubernetes cluster on ubuntu 18+. It's working fine. now i had added metric server but it's not working.

# kubectl get apiservices

v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   2d1h

    # kubectl describe apiservice v1beta1.metrics.k8s.io


    Message:               failing or missing response from https://10.106.145.77:443/apis/metrics.k8s.io/v1beta1: Get https://10.106.145.77:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.106.145.77:443: connect: connection refused
    Reason:                FailedDiscoveryCheck

I don't know why connection is refused. can any one help me or give me some hint to fix this. i have added RBAC, in cluster, does that is issue? I have try many solution from net but no one can help me. I had try to edit deployment yaml of metric server with args and insecure TLS but didn't help.

Other Details

# kubectl get all --all-namespaces | grep -i metrics-server

kube-system            pod/metrics-server-7f55d7ccbb-th9w9              1/1     Running   0          21s
kube-system            service/metrics-server              ClusterIP   10.106.145.77    <none>        443/TCP                                         26m
kube-system            deployment.apps/metrics-server              1/1     1            1           25m
kube-system            replicaset.apps/metrics-server-694db48df9              0         0         0       25m
kube-system            replicaset.apps/metrics-server-7f55d7ccbb              1         1         1       21s


# kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10

 - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        name: metrics-server
        ports:
        - containerPort: 4443
          hostPort: 4443

Yml File :-

# kubectl get -n kube-system deployment metr                                                                                                                     ics-server -o yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "2"
      creationTimestamp: "2020-01-29T14:49:06Z"
      generation: 2
      labels:
        k8s-app: metrics-server
      name: metrics-server
      namespace: kube-system
      resourceVersion: "951901"
      selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
      uid: 54137f75-af0a-45a5-a508-f4c38ee9ea25
    spec:
      progressDeadlineSeconds: 600
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          k8s-app: metrics-server
      strategy:
        rollingUpdate:
          maxSurge: 25%
          maxUnavailable: 25%
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            k8s-app: metrics-server
          name: metrics-server
        spec:
          containers:
          - args:
            - --cert-dir=/tmp
            - --secure-port=4443
            - --kubelet-insecure-tls
            - --kubelet-preferred-address-types=InternalIP
            image: k8s.gcr.io/metrics-server-amd64:v0.3.6
            imagePullPolicy: Always
            name: metrics-server
            ports:
            - containerPort: 4443
              hostPort: 4443
              name: main-port
              protocol: TCP
            resources: {}
            securityContext:
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              runAsUser: 1000
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /tmp
              name: tmp-dir
          dnsPolicy: ClusterFirst
          hostNetwork: true
          nodeSelector:
            beta.kubernetes.io/os: linux
            kubernetes.io/arch: amd64
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          serviceAccount: metrics-server
          serviceAccountName: metrics-server
          terminationGracePeriodSeconds: 30
          volumes:
          - emptyDir: {}
            name: tmp-dir
    status:
      availableReplicas: 1
      conditions:
      - lastTransitionTime: "2020-01-29T14:49:15Z"
        lastUpdateTime: "2020-01-29T14:49:15Z"
        message: Deployment has minimum availability.
        reason: MinimumReplicasAvailable
        status: "True"
        type: Available
      - lastTransitionTime: "2020-01-29T14:49:06Z"
        lastUpdateTime: "2020-01-29T15:14:26Z"
        message: ReplicaSet "metrics-server-7f55d7ccbb" has successfully progressed.
        reason: NewReplicaSetAvailable
        status: "True"
        type: Progressing
      observedGeneration: 2
      readyReplicas: 1
      replicas: 1
      updatedReplicas: 1

Answer 1

Find the args section and try this. Adding the command and /metrics server solved my problem along with updating the preferred address type and then restarting kubelet.

args:
        - --cert-dir=/tmp
        - --secure-port=4443
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

Answer 2

Faced similar issue with 503 Service Unavailable error message. Managed to fix the issue by making below changes.

On your components.yaml file, make sure cert path is correct:

-- cert-dir=/etc/kubernetes/pki

kubectl apply -f components.yaml

(change cert path to that instead of default /tmp. It may depend on your setup so, try find out where your pki certs are on your machine. Mine was on /etc/kubernetes/pki)