How to solve via Rancher a Kubernetes Ingress Controller Fake Certificate error

Question

I installed Rancher 2.6 on top of a kubernetes cluster. As cert-manager version I used 1.7.1.

helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.7.1 --set installCRDs=true --create-namespace


helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=MYDOMAIN.org \
--set bootstrapPassword=MYPASSWORD \
--set ingress.tls.source=letsEncrypt \    //<--- I use letsEncrypt
--set [email protected] \
--set letsEncrypt.ingress.class=nginx

After the installation was done, Rancher was successfully deployed on https:\mydomain.org.

LetsEncrypt SSL worked here fine. With Rancher I created a new RKE2 Cluster for my Apps.

So, I created a new Deployment for testing

• "rancher/hello-world:latest"
• 3x Replicas

Direct call of the nodeport ip adress with port, worked. http://XXXXXX:32599/

At this point I want to use a https subdomain hello.mydomain.org.

After study of documentation my approach was to create a new Ingress. I did it like you see on the following picture.

After creation of a new Ingress, I checked the section Ingresses of my hello world deployment. That new Ingress is now available there.

My expectation was that now I can go to **https://**hello.mydomain.org. But https doesn't work here, instead I got:

NET::ERR_CERT_AUTHORITY_INVALID
Subject: Kubernetes Ingress Controller Fake Certificate

Issuer: Kubernetes Ingress Controller Fake Certificate

Expires on: 03.09.2023

Current date: 03.09.2022

Where did I make a mistake? How to use LetsEncrypt for my deployments?

Answer 1

The fake certificate usually implies that the ingress controller is serving a default backend instead of what you expect it to. While a particular Ingress resource might be served over http as expected, the controller doesn't consider it servable over https. Most likely explanation is that the certificate is missing and ingress host isn't configured for https. When you installed rancher you only configured Rancher's own ingress. You need to setup certificates for each Ingress resource separately.

You didn't mention which ingress-controller you are using. With LE or other ACME based certificate issuers you'll usually need a Certificate Controller to manage certificate generation and renewal. I'd recommend cert-manager. There is an excellent tutorial for setting up LE, cert-manager and nginx-ingress-controller. If you're using Traefik, it is capable of generating LE certificates by itself, but the support is only partial in kubernetes environments (ie. no high availability), so your best bet is to use cert-manager even with that.

Even if or once you have set them up, cert-manager doesn't automatically issue certificates for every Ingress but only to those it is requested to. You need annotations for that.

With cert-manager, once you have set up the Issuer/ClusterIssuer and annotation, your ingress resource should look something like this (you can check the YAML from rancher):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
  name: hello-ingress
  namespace: hello-ns
spec:
  rules:
  - host: hello.example.com
    http:
      paths:
      - backend:
        service:
          name: hello-service
          port:
            number: 80
          path: /
            pathType: Prefix
      tls:
      - hosts:
        - hello.example.com
        secretName: hello-letsencrypt-cert

You might need to edit YAML directly and add spec.tls.secretName. If all is well, once you apply metadata.annotations and have set up spec.tls.hosts and spec.tls.secretName, the verification should happen soon and the ingress address should change to https://hello.example.com.

As a side note, I've experienced this issue also when the Ingress is behind a reverse proxy, such as HAproxy, and that reverse proxy (or Ingress) is not properly set up to use proxy protocol. You don't mention using one, but I'll write it just for the record.

If these steps don't solve your problem, you should check kubectl describe on the ingress and kubectl logs on the nginx-controller pods and see if anything stands out.

EDIT: I jumped to a conclusion, so I restructured this answer to also note the possibly of missing certificate manager altogether.