Providing traefik with nomad token from a Nomad variable

Question

I have set up a Nomad cluster with ACL enabled. I want to run a traefik job that uses Nomad's built-in service discovery. My traefik job looks like this (redacted some parts for brevity):

job "traefik" {

  group "traefik" {

    service {
      name = "traefik"
      provider = "nomad"
    }

    task "traefik" {
      driver = "docker"

      config {
        image = "traefik:v2.10.7"
        network_mode = "host"
        args = [
          "--api.dashboard=true",
          "--api.insecure=true",
          "--entrypoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
          "--providers.nomad=true",
          "--providers.nomad.exposedByDefault=false",
          "--providers.nomad.endpoint.address=http://10.0.0.3:4646/",
          "--providers.nomad.endpoint.token=<token>"
        ]
      }
    }
  }
}

The part I'm having trouble with is the --providers.nomad.endpoint.token argument passed to the traefik container. Since my cluster has ACLs enabled, I need to provide a Nomad token here, otherwise I get this error:

Provider connection error failed to load initial nomad services: Unexpected response code: 403 (Permission denied)

I have created a token in my cluster for Traefik to use (and when I hard-code it in the job specification, it works), and I have added that token to a Nomad variable with the path nomad/jobs/traefik, and the name NOMAD_TOKEN.

What I can't figure out is how I can change my job specification to inject this variable into the --providers.nomad.endpoint.token. Essentially I want something like this:

--providers.nomad.endpoint.token=:${NOMAD_TOKEN}

I think I have to use a template block, but I don't know how to do that with docker args?

Or is there a better way to provide traefik with the Nomad token?

Answer 1

how I can change my job specification

and I have added that token to a Nomad variable with the path nomad/jobs/traefik,

Strings inside config.args are not templated with Go Nomad template. There is no way to do that inside config.args.

We need some wrapper script that can be Nomad templated and run that wrapper script. This requires checking traefik docker image what is has inside entrypoint so you'll run the same from the script. Something along:

task "traefik" {
  driver = "docker"
  config {
    image = "traefik:v2.10.7"
    network_mode = "host"
    command = "sh"
    args = ["-xc", "/local/script.sh"]
  }
  template {
    destination = "/local/script.sh"
    data = <<EOF
      #!/bin/sh
      traefik \
        --api.dashboard=true \
        --api.insecure=true \
        --entrypoints.web.address=:{{env "NOMAD_PORT_http"}} \
        --entrypoints.traefik.address=:{{env "NOMAD_PORT_admin"}} \
        --providers.nomad=true \
        --providers.nomad.exposedByDefault=false \
        --providers.nomad.endpoint.address=http://10.0.0.3:4646/ \
        --providers.nomad.endpoint.token={{with nomadVar "nomad/jobs/traefik"}}{{.THE_TOKEN}}{{end}}
    EOF
  }
}

Overall, I do not like the idea of inline, long configuration files. I would rather just create the traefik configuration file. It would look like the following, also untested, see documentation:

# traefik.yaml
---
api:
   dashboard: true
   insecure: true
entrypoints:
  web:
    address: :{{env "NOMAD_PORT_http"}}
  traefik: 
    address: :{{env "NOMAD_PORT_admin"}}
providers:
   nomad:
      exposedByDefault: false
      endpoint:
        address: http://10.0.0.3:4646/
        token: {{with nomadVar "nomad/jobs/traefik"}}{{.THE_TOKEN}}{{end}}

And nomad job would just take the configuration file to template it. The file("./traefik.yaml") path is relative to the directory you are running nomad run from .

   task "traefik" {
      driver = "docker"
      config {
        image = "traefik:v2.10.7"
        network_mode = "host"
        args = ["--configFile=/local/traefik.yaml"]
      }
      template {
        destination = "/local/traefik.yaml"
        data = file("./traefik.yaml")
      }
    }

Answer 2

After some help from somebody from the Nomad team, I was able to do it very simply via an identity block. My traefik job spec now looks like this:

job "traefik" {

  group "traefik" {

    service {
      name = "traefik"
      provider = "nomad"
    }

    task "traefik" {
      driver = "docker"

      identity {
        env           = true
        change_mode   = "restart"
      }

      config {
        image = "traefik:v2.10.7"
        network_mode = "host"
        args = [
          "--api.dashboard=true",
          "--api.insecure=true",
          "--entrypoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
          "--providers.nomad=true",
          "--providers.nomad.exposedByDefault=false",
          "--providers.nomad.endpoint.address=http://10.0.0.3:4646/",
        ]
      }
    }
  }
}