Using a custom certificate for the Kubernetes api server with minikube

Question

I have been trying to find how to do this but so far have found nothing, I am quite new to Kubernetes so I might just have looked over it. I want to use my own certificate for the Kubernetes API server, is this possible? And if so, can someone perhaps give me a link?

Answer 1

Ok, so here is my idea. We know we cannot change cluster certs, but there is other way to do it. We should be able to proxy through ingress.

First we enabled ingres addon:

➜  ~ minikube addons enable ingress

Given tls.crt and tls.key we create a secret (you don't need to do this if you are using certmanager but this requires some additinal steps I am not going to describe here):

➜  ~ kubectl create secret tls my-tls --cert=tls.crt --key tls.key

and an ingress object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-k8s
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - foo.bar.com
    secretName: my-tls
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes
            port:
              number: 443

Notice what docs say about CN and FQDN: k8s docs:

Referencing this secret in an Ingress tells the Ingress controller to secure the channel from the client to the load balancer using TLS. You need to make sure the TLS secret you created came from a certificate that contains a Common Name (CN), also known as a Fully Qualified Domain Name (FQDN) for https-example.foo.com.

The only issue with this approach is that we cannot use certificates for authentication when accessing from the outside.

But we can use tokens. Here is a page in k8s docs: https://kubernetes.io/docs/reference/access-authn-authz/authentication/ that lists all possible methods of authentication.

For testing I choose serviceaccout token but feel free to experiment with others.

Let's create a service account, bind a role to it, and try to access the cluster:

➜  ~ kubectl create sa cadmin
serviceaccount/cadmin created
➜  ~ kubectl create clusterrolebinding --clusterrole cluster-admin --serviceaccount default:cadmin cadminbinding
clusterrolebinding.rbac.authorization.k8s.io/cadminbinding created

Now we follow these instructions: access-cluster-api from docs to try to access the cluster with sa token.

➜  ~ APISERVER=https://$(minikube ip)
➜  ~ TOKEN=$(kubectl get secret $(kubectl get serviceaccount cadmin -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode )
➜  ~ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure -H "Host: foo.bar.com"
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.39.210:8443"
    }
  ]
}

note: I am testing it with invalid/selfsigned certificates and I don't own the foo.bar.com domain so I need to pass Host header by hand. For you it may look a bit different, so don't just copypate; try to understand what's happening and adjust it. If you have a domain you should be able to access it directly (no $(minikube ip) necessary).

As you should see, it worked! We got a valid response from api server.

But we probably don't want to use curl to access k8s.

Let's create a kubeconfig with the token.

kubectl config set-credentials cadmin --token $TOKEN --kubeconfig my-config
kubectl config set-cluster mini --kubeconfig my-config --server https://foo.bar.com
kubectl config set-context mini --kubeconfig my-config --cluster mini --user cadmin
kubectl config use-context --kubeconfig my-config mini

And now we can access k8s with this config:

➜  ~ kubectl get po --kubeconfig my-config
No resources found in default namespace.

Answer 2

Yes, you can use your own certificate and set inn the Kubernetes API server.

Suppose you have created the certificate move and save them to specific node directory:

sudo mkdir -p /var/lib/kubernetes/

sudo mv ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml /var/lib/kubernetes/

The instance internal IP address will be used to advertise the API Server to members of the cluster. Get the internal IP:

INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)

you can crate the service of API server and set it.

Note : Above mentioned example is specifically with consider the GCP instances so you might have to change some commands like.

INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
      http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip) 

for the above command, you can provide the manual bare metal IP list instead of getting from GCP instance API if you are not using it.

Here we go please refer to this link : https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/08-bootstrapping-kubernetes-controllers.md#configure-the-kubernetes-api-server

here you can find all the details for creating and setting whole Kubernetes cluster from scratch along will detailed document and commands : https://github.com/kelseyhightower/kubernetes-the-hard-way