How to handle authentication on Prometheus metrics endpoint for an application on GKE?

Question

I have a set of microservices running in a GKE cluster. They all have a /prometheus endpoint which exposes metrics for Managed Prometheus collectors to collect.

Everything works fine when the endpoint is exposed and I follow the instructions here: https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed - the metrics get collected and are visible under Cloud Monitoring.

However, I would like to protect this endpoint with some sort of authorization on the app level so that not anyone can access this endpoint for obvious reasons.

How can I do this? I don't see any option to give the PodMonitoring scraper some token or key to use when making http requests. What is the recommended approach?

Answer 1

Accessing and analyzing ingested data can be done through the Prometheus UI. Based on the metrics scope associated with your project, this UI executes PromQL queries against all of the data in your Google Cloud project.

In addition, the user interface serves as an authentication proxy for accessing ingested data. This feature can be used for client tools that don't support OAuth2, including Grafana. You must also deploy the Prometheus UI if you intend to use Grafana to visualize data from Managed Service for Prometheus.

To deploy prometheus UI you can follow this documentation.

To Authenticate the UI you can follow this documentation.