Certmanager Cross Project DNS01 Challenge Fail

Question

Project Prod and Project Staging have been setup and each running a GKE cluster. Cert-manager is installed to automate the process of certificate issuance as explained in official docs.

Project Prod has DNS that maps to both prod and staging cluster istio gateway IP addresses.

On DNS01 challenge for cluster in Project Prod, manages to authenticate, and certificate is issued successfully.

But the cluster running in Project Staging, fails to get certificate due to not enough permission to authenticate and verify via Cloud DNS setup in Project Prod.

In Project Prod, there is a service account with dns/admin role that is setup via GKE secret and accessed in clusterissuer like so

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-clusterissuer
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod-clusterissuer
    solvers:
    # ACME DNS-01 provider configurations
    - dns01:
        # Google Cloud DNS
        cloudDNS:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: cloud-dns-key
            key: key.json
          # The project in which to update the DNS zone
          project: iprocure-server-prod

Certificate is issued successfully in Project Prod GKE cluster.

Project Staging GKE cluster, clusterissuer has its service account with dns/admin role just like in Project Prod, but fails to perform dns01 challenge in Project Prod DNS.

Following error is seen when kubectl describe challenge

Type     Reason        Age                     From          Message
----     ------        ----                    ----          -------
Warning  PresentError  2m56s (x19 over 7h14m)  cert-manager  Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden

What should be done to Project Staging service account to enable dns01 challenge to be performed in Project Prod Clous DNS

Answer 1

I faced this problem too. I had to replicate the service account in my Prod Project with DNS ADMIN permission to Staging Project so that the GKE cluster in Staging can have enough permission to authenticate and verify via Cloud DNS setup in the Prod project

You have to create a SA in Prod Project with DNS ADMIN permission and master the email of that SA then go to project B and make that SA email a member by adding it as a member with also DNS ADMIN permission.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: {{ .Values.app.certificate.issuer.name }}
  namespace: {{ .Values.app.namespace }}
  labels:
    app.kubernetes.io/managed-by: "Helm"
spec:
  acme:
    email: {{ .Values.app.certificate.acme.email }}
    privateKeySecretRef:
      name: {{ .Values.app.certificate.issuer.name }}
    server: {{ .Values.app.certificate.acme.server }}
    solvers:
      - dns01:
          cloudDNS:
            project: {{ .Values.app.project_id }} ## Make sure this is the project of where DNS is e.g Prod Project
            serviceAccountSecretRef:
              name: {{ .Values.secrets.name }}
              # Secret from the google service account key
              key: key.json