How set up Prometheus Operator with Grafana to enable basic Kubernetes monitoring

Question

I followed a bunch of tutorials on how to monitor Kubernetes with prometheus and Grafana All referring to a deprecated helm operator

According to the tutorials Grafana comes out of the box complete with cluster monitoring. In practice Grafana is not installed with the chart

helm install prometheus-operator stable/prometheus -n monitor

nor is it installed with the newer community repo

helm install prometheus-operator prometheus-community/prometheus -n monitor

I installed the Grafana chart independently

helm install grafana-operator grafana/grafana -n monitor

And through the UI tried to connect using inner cluster URLs

prometheus-operator-server.monitor.svc.cluster.local:80

prometheus-operator-alertmanager.monitor.svc.cluster.local:80

the UI test indicates success but produces no metrics.

Is there a ready made Helm operator with out of the box Grafana? How can Grafana interact with Prometeus?

Answer 1

You've used the wrong charts. Currently the project is named kube-prometheus-stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

If you look at values.yaml you'll notice switches for everything, including prometheus, all the exporters, grafana, all the standard dashboards, alerts for kubernetes and so on. It's all installed by one chart. And it's all linked together out of the box.

They only additional thing you might need is an Ingress/ELB for grafana, prometheus, and alertmanager to be able to open them without port-forwarding (don't forget to add ouath2-proxy or smth else cause it's all opened with no password by default).

Answer 2

I wouldn't bother, look at PaaS like Datadog, NewRelic etc. What you are describing becomes a costly nightmare at scale. It's just not worth the hassle for what you get ihmo.