Upgrading kube-prometheus-stack via Helm to chart v56.2.1 fails on Grafana with Sensitive key error

Question

I recently tried upgrading kube-prometheus-stack on my AWS EKS Kubernetes cluster via Helm to chart v56.2.1 using Terraform and it failed while upgrading Grafana with the following error:

Sensitive key ‘auth.generic_oauth.client_secret’ should not be defined explicitly in values. Use variable expansion instead.

I'm not specifying any client_secret via values.yml file for that Helm chart but we are using AzureAD for authentication.

How to fix this?

Answer 1

There are 2 possible solutions to fix this issue:

1. Set grafana.assertNoLeakedSecrets to false in the values.yml file.

2. If auth.generic.oauth.client_secret is specified in the values.yml file, remove it from that file and set GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as the environment variable for Grafana.

References:

• Grafana - Sensitive Key Known Issue
• Grafana - Override configuration with environment variables

Answer 2

resource "kubernetes_secret" "grafana" {
  ...
  data = {
    "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET" = var.auth_generic_oauth_client_secret
  }
}


resource "helm_release" "kube_prometheus_stack" {
  chart = "kube-prometheus-stack"
  ...
  set {
    name  = "grafana.envFromSecret"
    value = kubernetes_secret.grafana.metadata[0].name
  }
}

Grafana's Helm chart provides multiple ways to securely mount environment variables: envFromSecret, envRenderSecret, envFromSecrets, envFromConfigMaps, and extraSecretMounts. You can find more information at charts/grafana/values.yaml.

The variable name is in the format pf GF_<SectionName>_<KeyName>. As described at grafana/variable-expansion. So:

[auth.google]
client_secret = ...

becomes GF_AUTH_GOOGLE_CLIENT_SECRET. Any env variable set will by default override the value if also defined in the grafana.ini. If you would like to use the $__file provider, you can use the extraSecretMounts and set the following helm values:

resource "kubernetes_secret" "auth_generic_oauth_secret" {
  ...
  metadata {
    name = "auth-generic-oauth-secret"
  }
  data = {
    "client_id" = "......"
    "client_secret" = "......."
  }
}

resource "helm_release" "kube_prometheus_stack" {
  ...
  set { 
    name  = "grafana.extraSecretMounts[0].name"
    value = "auth-generic-oauth-secret-mount"
  }
  set {
    name  = "grafana.extraSecretMounts[0].secretName"
    value = kubernetes_secret.auth_generic_oauth_secret.metadata[0].name
  }
  set { 
    name  = "grafana.extraSecretMounts[0].defaultMode"
    value = 0440
  }
  set { 
    name  = "grafana.extraSecretMounts[0].mountPath"
    value = "/etc/secrets/auth_generic_oauth"
  }
  set {
    name  = "grafana.extraSecretMounts[0].readOnly"
    value = true
  }
}

Now that the secret is mounted as a file, we can use the $__file provider to set the credentials:

  set {
    name = "grafana.grafana\\.ini.auth.generic_oauth.client_id"
    value = "$__file{/etc/secrets/auth_generic_oauth/client_id}"
  }
  set {
    name = "grafana.grafana\\.ini.auth.generic_oauth.client_secret"
    value = "$__file{/etc/secrets/auth_generic_oauth/client_secret}"
  }