Kubernetes liveness probe on a secure mTLS health check endpoint

Question

I would like some help to solve a particular Kubernetes + mTLS problem please.

How to make Kubernetes liveness probe work on a secure https mTLS health check endpoint please?

My application is a web application where a health check endpoint is exposed over a particular port, same port as other business endpoints.

Per security, audit and compliance review, I must secure all my endpoints over mTLS, even the simple and innocent health check endpoint.

Per security, audit and compliance review, I cannot expose any other ports like doing my business endpoints on https port 1, but health on http port 2.

Hence, this is failing and marking my app as down (since it is over simple http, the endpoint is https)

          livenessProbe:
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10

Just to confirm during a test, we disabled https and mTLS, enabling plain old http, everything is working perfectly, but we simply cannot do that.

How to solve this problem please?

Thank you.

Answer 1

Can you try changing scheme: HTTP to HTTPS?

livenessProbe: httpGet: path: /health port: 8080 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10

If scheme field is set to HTTPS, the kubelet sends an HTTPS request skipping the certificate verification.

Answer 2

1. Firstly, can you confirm if there are any operations or port 8080 is in use? (Check deployment or pod yaml file and make sure you have referenced it). If nothing is going on over there your health checks would fail
2. Can you remove the "scheme" part set to HTTP? HTTP is the default scheme so it's redundant, unless you want to change it to HTTPS
3. path: /health needs to be valid, make sure it exists on the application.

Answer 3

Mutual TLS (mTLS) aims to warrant the authenticity of both end, this means the probing side (kubelet) must have access to the keys and certificates that it uses to identify itself as well as others at runtime. Kubernetes doesn't have this built-in, a well known solution to this is here. There are many service mesh solutions out there embedded with this solution if you do not want to deploy it directly. Istio, Nginx Service Mesh, AWS App Mesh... just to name a few.

Answer 4

You can use script for your Readiness probe. inside that script you can simple cURL on the end point and also provide required certificate and CA cert.

For example:

curl -k https://<url>/health -v –key key.pem –cacert ca.pem –cert client.pem

I assume certificate should be stored in secret in the same namespace. So you can mount secret having cert in your pod so that your script can use those path to access certificates.

Answer 5

I'm investigating a solution for the same problem. What I'm planning to do is to have a trusted certificate inside the pod and instead of using http probe, use a command probe and curl my healthcheck endpoint from inside the pod using said certificate.

Answer 6

If you don't want to follow above approach then can open one port inside pod which will not be exposed through service. it will be internal to pod and can be used for readiness or liveness purpose. Microservices follow these approach now a days.

Answer 7

similar to https://stackoverflow.com/a/67047770/1747983 answer, here a full YAML example.

I couldn't figure out how to use variables (without going to the env). Would be nice for the PW , port and path.

        readinessProbe:
          exec:
            command: ["/bin/sh", "-c", "curl -sS -I -k --cert-type P12 --cert /myPath/my-keystore.p12:MyPass123 -f https://localhost:12002/component/info"]
          initialDelaySeconds: 20
          periodSeconds: 5
        livenessProbe:
          exec:
            command: ["/bin/sh", "-c", "curl -sS -I -k --cert-type P12 --cert /myPath/my-keystore.p12:MyPass123 -f https://localhost:12002/component/info"]
          initialDelaySeconds: 20
          periodSeconds: 5
          timeoutSeconds: 5