gmp google managed prometheus example not working on a brand new vanilla stable gke autopilot cluster

Question

Google managed prometheus seems like a great service however at the moment it does not work even in the example... https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed

Setup:

• create a new autopilot cluster 1.21.12-gke.2200
• enable manage prometheus via gcloud cli command gcloud beta container clusters update <mycluster> --enable-managed-prometheus --region us-central1
• add port 8443 firewall webhook command
• install ingress-nginx
• try and use the PodMonitoring manifest to get metrics from ingress-nginx

Error from server (InternalError): error when creating "ingress-nginx/metrics.yaml": Internal error occurred: failed calling webhook "default.podmonitorings.gmp-operator.gke-gmp-system.monitoring.googleapis.com": Post "https://gmp-operator.gke-gmp-system.svc:443/default/monitoring.googleapis.com/v1/podmonitorings?timeout=10s": x509: certificate is valid for gmp-operator, gmp-operator.gmp-system, gmp-operator.gmp-system.svc, not gmp-operator.gke-gmp-system.svc

There is a thread suggesting this will all be fixed this week (8/11/2022), https://github.com/GoogleCloudPlatform/prometheus-engine/issues/300, but it seems like this should work regardless.

if I try to port forward ...

kubectl -n gke-gmp-system port-forward svc/gmp-operator 8443
error: Pod 'gmp-operator-67d5fff8b9-p4n7t' does not have a named port 'webhook'

Answer 1

This is now fixed, if you have trouble this permalink how to guide includes troubleshooting tips. It's GMP (google managed prometheus) on GKE Autopilot, I just ran through their examples, and they more or less work. https://gist.github.com/neoakris/bfc67fef83fecde6384ab4f8afe5fc95