How Do I Properly Set --kubelet-certificate-authority apiserver parameter?

Question

I am using KubeSpray to provision a two node cluster on AWS. By default, the --kubelet-certificate-authority parameter is not used. However, I would like to set it.

I do not know the correct setting for --kubelet-certificate-authority. When I set it to /etc/kubernetes/pki/ca.crt I see messages like the following in my logs:

TLS handshake error from 10.245.207.223:59278: remote error: tls: unknown certificate authority

In order to create an isolated environment for testing, I SSH'ed to my controller node to run this command which runs apiserver on the non-standard port of 4667. I copied these values directly from /etc/kubernetes/manifests/kube-apiserver.yaml. You'll need to adjust the value to match your own cluster. I purposely am running the container in interactive mode so that I can see the log messages.

sudo docker run \
  --name api-server-playground \
  -it \
  --rm \
  --network host \
  --volume /etc/kubernetes:/etc/kubernetes:ro \
  --volume /etc/pki:/etc/pki:ro \
  --volume /etc/ssl:/etc/ssl/:ro \
  k8s.gcr.io/kube-apiserver:v1.18.9 \
  kube-apiserver \
    --advertise-address=10.245.207.223 \
    --allow-privileged=true \
    --anonymous-auth=True \
    --apiserver-count=1 \
    --authorization-mode=Node,RBAC \
    --bind-address=0.0.0.0 \
    --client-ca-file=/etc/kubernetes/ssl/ca.crt \
    --cloud-config=/etc/kubernetes/cloud_config \
    --cloud-provider=aws \
    --enable-admission-plugins=NodeRestriction \
    --enable-aggregator-routing=False \
    --enable-bootstrap-token-auth=true \
    --endpoint-reconciler-type=lease \
    --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
    --etcd-certfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal.pem \
    --etcd-keyfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal-key.pem \
    --etcd-servers=https://10.245.207.119:2379 \
    --event-ttl=1h0m0s \
    --insecure-port=0 \
    --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt \
    --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key \
    --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP \
    --profiling=False \
    --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt \
    --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key \
    --request-timeout=1m0s \
    --requestheader-allowed-names=front-proxy-client \
    --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --secure-port=6447 \
    --service-account-key-file=/etc/kubernetes/ssl/sa.pub \
    --service-cluster-ip-range=10.233.0.0/18 \
    --service-node-port-range=30000-32767 \
    --storage-backend=etcd3 \
    --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt \
    --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key

Now it's possible to SSH into the controller again and use curl to interact with the custom apiserver container.

• Set the APISERVER variable to point to the controller node. I'm using the value of the advertise-address parameter above.

APISERVER=https://10.245.207.223:6447

• Get a token.

TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode); echo "TOKEN=$TOKEN"

• Make an api request. This request will fail because "Peer's Certificate issuer is not recognized.".

curl --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

An error message will appear in the docker log. It will look like this:

I0921 14:39:07.662368       1 log.go:172] http: TLS handshake error 
from 10.245.207.223:59278: remote error: tls: unknown certificate authority

• Now use the -k curl parameter to bypass the recognition issue.

curl -k --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.245.207.223:6447"
    }
  ]
}

You'll see the request works correctly. However, I don't want to use the -k parameter. So I tried to use the certificate authority from the apiserver.

• Get the certificate authority from the apiserver.

echo | \
    openssl s_client -connect $APISERVER 2>/dev/null | \
    openssl x509 -text | \
    sed -n "/BEGIN CERTIFICATE/,/END CERTIFICATE/p" \
    > apiserver.ca.crt

• Use the certificate authority file for the api request.

curl --cacert apiserver.ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

UPDATE

Following a thought prompted by Wiktor's response, I added /etc/kubernetes/ssl/ca.crt as the certificate authority. And used that file in my curl command.

curl --cacert /etc/kubernetes/ssl/ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

This worked.

Answer 1

In order to make the --kubelet-certificate-authority flag work you first need to make sure you got Kubelet authentication and Kubelet authorization enabled. After that you can follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelet. And finally, you can edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.

So, to sum up the steps to do are:

1. Kubelet authentication:

• start the kubelet with the --anonymous-auth=false flag

• start the kubelet with the --client-ca-file flag, providing a CA bundle to verify client certificates with

• start the apiserver with --kubelet-client-certificate and --kubelet-client-key flags

• ensure the authentication.k8s.io/v1beta1 API group is enabled in the API server

• start the kubelet with the --authentication-token-webhook and --kubeconfig flags

• the kubelet calls the TokenReview API on the configured API server to determine user information from bearer tokens

2. Kubelet authorization:

• ensure the authorization.k8s.io/v1beta1 API group is enabled in the API server

• start the kubelet with the --authorization-mode=Webhook and the --kubeconfig flags

• the kubelet calls the SubjectAccessReview API on the configured API server to determine whether each request is authorized

3. Use the --kubelet-certificate-authority flag to provide the apiserver with a root certificate bundle to use to verify the kubelet's serving certificate.

More details can be found in the linked documentation.