unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key

Question

I have a pfx certificate stored as secret in azurekeyvault as you can see

I have used akv2k8s tool to create kubernetes secret for accessing azurekeyvault in azure kubernetes cluster. the kubernetes secret got created successfully as you can see

but as i applied it to the ingress.yaml it's still using the kubernetes fake certificate, when i checked the logs of nginx-ingress pod i got to see the following error as you can see,

and this is my ingress-srv.yaml file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-srv
  annotations:
    kubernetes.io/ingress.class: nginx
    
    
spec:
  tls:
  - hosts:
    - devc.leverauto.com
    secretName: my-pfx-cert-secret-from-akv
  defaultBackend:
    service:
      name: lever-webapp-service
      port:
        number: 3000
  rules:
  - host: devc.leverauto.com
    http:
      paths:
      - path: /auth/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-authentication-service
            port:
              number: 5000
      - path: /auction/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-auction-service
            port:
              number: 5010 
      - path: /audit/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-audits-service
            port:
              number: 5005                      
      - path: /inventory/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-inventory-service
            port:
              number: 5006         
      - path: /lender/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-lender-service
            port:
              number: 5009 
      - path: /payment/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-payments-service
            port:
              number: 5002
      - path: /report/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-reports-service
            port:
              number: 5003 
      - path: /sopraData/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-sopra-datapull-service
            port:
              number: 5011
      - path: /sopra/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-sopra-service
            port:
              number: 5008
      - path: /vehicle/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-vehicle-service
            port:
              number: 5001
      - path: /workflow/api/v1/
        pathType: Prefix
        backend:
          service:
            name: lever-workflow-service
            port:
              number: 5004  
      - path: /
        pathType: Prefix
        backend:
          service:
            name: lever-webapp-service
            port:
              number: 3000    

                                              

can't find how to solve this?

Answer 1

• You are getting this error because the relevant ingress rules might not be specifying a matching hostname to the certificate that is uploaded as a secret in the keyvault. Also, since you are configuring a NGINX server, the ingress controller provides the flag ‘—default-ssl-certificate’. The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided, NGINX will use a self-signed certificate and at that time, a ‘foo-tls’ should be added in the ‘nginx-controller’ deployment like ‘--default-ssl-certificate=default/foo-tls’.

• Also, please ensure whether you have enabled the ‘SSL Passthrough’ feature which is disabled by default as this is required to enable passthrough backends in Ingress objects. Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints. This can also be a problem in the way your secret in the certificate is added in the keyvault.

• Please check whether HSTS security feature is disabled or not as it is enabled by default since it is an opt-in security enhancement specified using special response header. To disable this behaviour, please use ‘hsts : false’ in the configuration while deploying.

• Finally, please check whether server-side HTTPS enforcement through redirect is disabled or not as when enabled, the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. It can be disabled globally by using ‘ssl-redirect : false’ in the NGINX yaml configuration with the annotation as ‘nginx.ingress.kubernetes.io/ssl-redirect: "false"’.

For more detailed information regarding the secret identification and unrecognition in the Azure key vault, kindly refer to the link below: -

https://kubernetes.github.io/ingress-nginx/user-guide/tls/

Answer 2

I was able to get through this problem by some temporary solution, my ssl certificate provider is GoDaddy and it is stored as pfx secret in azure key vault. with akv2k8s i tried to create a kubernetes secret from the secret in keyvault, the kubernetes secret got created but as i have described above it got me the issue that public key not matching.

so how i solved this is we had another instance of the application running inside azure vm and we imported the same ssl certificate there and i thought that maybe somehow the ssl in keyvault got corrupted but i know that certificate in vm had no problem because we were accessing that vm domain over the internet and it was showing valid certificate, so i exported the certificate from vm with a password and uploaded it to keyvault as certificate and then again used akv2k8s to create kubernetes secret it got created successfully and i didn't face the above issue ingress pods applied the ssl to kubernetes domain successfully.

Now as you can see in above screenshot my ssl got expired on 15th of july 2022 and the ssl was on autorenewal so it got updated in the keyvault. so now i need to update ssl to all instances again. so this time i thought to try and create kubernetes secret using secret in azurekeyvault as i know this time there is no issue with secret in keyvault as i have imported the same in vm it got updated to vm domain and working fine.

But i again faced same issue that public key not matching, now again i tried to export the new certificate from the vm with a password and uploaded it as certificate to key vault and used akv2k8s to create kubernetes secret and again ingress applied this successfully to kubernetes domain. so this is just temporary solution but it solved my problem.