Pattern for deploying dashboards using kube prometheus stack's grafana dashboard sidecar

Question

I am trying to get kube prometheus stack's grafana dashboard sidecar to deploy dashboards depending on the environment (dev, staging, prod, all environments). I can't quite figure out a good way to do this.

My folder structure currently looks like such:

kube-prometheus-stack
  charts
    grafana
      templates
        env
          dev
            dashboard1-cm.yaml
            dashboard2-cm.yaml
          staging
          prod
          all-environments

I'd like to deploy dev dashboards to dev, staging to staging and so on.

I've been browsing https://github.com/kiwigrid/k8s-sidecar but haven't been able to figure out a good pattern to handle this.

Curious how others are doing this.

Answer 1

I have tried just one simple thing:

1. add one more folder for env-specific folder only

2. add the placement for the specific location of the dashboards in that folder to the specific value-env.yaml

   grafana:
       sidecar:
         dashboards:
           enabled: true
           label: grafana_dashboard
           annotations: {}
           multicluster: false
           alpha: true
   

3. In yaml associated with specific dashboard (mqtt2.yaml) I have

   {{- if $.Values.grafana.sidecar.dashboards.alpha }}
    ....
   {{- end }}
   

surronding previos config to exclude it if .alpha property isnt set

4. upgrading helm for the specific env with passing the specific config for that:

   helm -n monitoring upgrade prometheus ./ -f values-core.yaml -f values-alpha.yaml --recreate-pods
   

For me it works and now I'm able ho have different dashboards at the different envs. Also note, dashboards-1.14 folder still have all dashboards being still deployed even after the sidecar configuration update, so move all dashboards from just dashboards to it, while have set up other folders to store data instead of generic just dashboards to some env-specific like mine dashboards-alpha.