How to get k8s controller manager's metrics?

Question

I have deploy a k8s cluster with kubeadm, I want to get controller manager's metrics with following command:

curl -k https://localhost:10257/metrics

but got the following error:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

So my question is, how to get k8s controller manager's metrics?

Answer 1

This is a forbidden error due to permission issues which need to be authenticated with a valid user. For this,You need to create a service account, then give that service account access permissions to the metrics Path through RBAC, then this will make that service account to get the metrics.

As per this Role and Cluster Binding doc, you need to allow metrics path(replace with /healthz) as below and give a try.

Allow GET and POST requests to the non-resource endpoint /healthz and all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding to be effective):
rules:


- nonResourceURLs: ["/healthz", "/healthz/*"] # '*' in a nonResourceURL is a suffix glob match


 verbs: ["get", "post"]

Answer 2

If your cluster uses RBAC, reading metrics requires authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing /metrics. To get the kubernetes controller manager, scheduler and other control plane components you have to do the authorized user. As per this Kubernetes System Components Metrics.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get