Unable to access prometheus endpoint using the ip address after deployment using serviceaccount and rbac

Question

I am using this link to set up Prometheus using servicemonitoring and RBAC rules.

After deployment, I am trying to access prometheus endpoint using http://<ipaddress_server>:32001 to display the metrics.

I am also trying to add the prometheus datasource in grafana ( running on same node) as http://localhost:32001 or http://localhost:9090, but both didn't work.

$ kubectl get svc -A | grep prom
default                prometheus                                 NodePort       10.XX.XX.XX   <none>         9090:32001/TCP                        14m
default                prometheus-operated                        ClusterIP      None             <none>         9090/TCP                              14m
default                prometheus-operator                        ClusterIP      None             <none>         8080/TCP                              15m

kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 3h7m

kubectl get deployment
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator   1/1     1            1           3h7m


kubectl get pods -A | grep prom
default                prometheus-operator-5dc8bc6f9c-9lwtw                              1/1     Running             0                 3h7m
default                prometheus-prometheus-0                                           2/2     Running             0                 3h6m
default                prometheus-prometheus-1                                           2/2     Running    

kubectl get servicemonitor
NAME              AGE
prometheus-self   3h4m
     0                 3h6m

I also tried port forwarding using below command:

kubectl port-forward svc/prometheus 9090

But, it doesn't show anything.

Please help, as I am new to kubernetes and also prometheus and grafana.

Answer 1

Try to create ConfigMap with following Configuration and Apply it will work.

ConfigMap is generally used to provide Runtime Configurations to our Applications(Pods) but make sure you shouldn't use ConfigMap to provide credentials, use Secrets instead.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  prometheus.yaml: |-
   {
    "apiVersion": 1,
    "datasources": [
        {
           "access":"proxy",
            "editable": true,
            "name": "prometheus",
            "orgId": 1,
            "type": "prometheus",
            "url": "http://prometheus-operated.default.svc:9090",
            "version": 1
        },
    ]
}

Here What I tried instead of using localhost:9090 used service name and namespace name along with Port number as Host DNS.

"http://serviceName.NamespaceName.svc:port"

But you can't access above URL in browser.

After Applying this ConfigMap you can check your Grafana Dashboard by Port-Forwarding kubectl port-forward svc/grafana 3000

You can Follow this https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/grafana-dashboardSources.yaml