Trigger Kubernetes job on secret update

Question

How do you trigger rerunning a job once one of its dependent secrets get updated?

For example, say I have this simple job:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-test
spec:
  template:
    metadata:
      labels:
        app: databricks
    spec:
      containers:
      - name: job-test
        image: alpine
        command:
          - echo
          - hello $(DAN_TEST)
        env:
          - name: DAN_TEST
            valueFrom:
              secretKeyRef:
                name: dan-test
                key: dan-test-1
      restartPolicy: Never
  backoffLimit: 4

Adding this job makes it run and print out the secret, but when the secret is changed, the job is not automatically rerun.

Is there built-in or 3rd party extension resource that can target the secret and the job and trigger a rerun?

Answer 1

stakater/Reloader allows resources to be reloaded based on watched changes in a ConfigMap or Secret.

Unfortunately, Job and CronJob resources are not supported, so to make it work I used a Deployment with an initContainer which executed the job, and a container which kept the pod alive.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: job-test
  labels:
    app: job-test
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: job-test
  template:
    metadata:
      labels:
        app: job-test
    spec:
      initContainers:
        - name: job-test
          image: alpine
          imagePullPolicy: Always
          command:
            - echo
            - hello $(DAN_TEST)
          env:
            - name: DAN_TEST
              valueFrom:
                secretKeyRef:
                  name: dan-test
                  key: dan-test-1
      containers:
        - name: job-test-wait
          image: alpine
          command:
            - tail
            - -f
            - /dev/null

I forgot exactly why the pod needed to be kept alive, but from trial and error this is what I got working a few months ago and am still using today.

Answer 2

I think it's maybe better to use a ConfigMap instead of a Secret, because ConfigMaps are designed to be more easily updated and can be monitored for changes using Kubernetes' built-in watch functionality!

let me update your code again :

apiVersion: batch/v1
kind: Job
metadata:
  name: job-test
  annotations:
    configmap.reloader.stakater.com/reload: "dan-test"
spec:
  template:
    metadata:
      labels:
        app: databricks
    spec:
      containers:
      - name: job-test
        image: alpine
        command:
          - echo
          - hello $(DAN_TEST)
        env:
          - name: DAN_TEST
            valueFrom:
              configMapKeyRef:
                name: dan-test
                key: dan-test-1
      restartPolicy: Never
  backoffLimit: 4

as you can see,I'm using the "configmap.reloader.stakater.com/reload" annotation to tell Kubernetes to monitor the "dan-test" ConfigMap for changes. and about third-party tools, the "Kubernetes Event-Driven Autoscaling" (KEDA) project includes a "Secret" scaler that can trigger a Job based on changes to a Secret or other tools like Argo Workflows also provide similar functionality.

update :
Ok I think to automatically trigger a rerun of a Job when a ConfigMap or Secret it depends on is updated, you can use the cronjob resource instead of the job resource

Let me explain cronjob to you with this example :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job-test
spec:
  schedule: "*/1 * * * *" # run every minute
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: job-test
            image: alpine
            command:
              - echo
              - hello $(DAN_TEST)
            env:
            - name: DAN_TEST
              valueFrom:
                configMapKeyRef:
                  name: dan-test
                  key: dan-test-1
          restartPolicy: OnFailure

As you see,restartPolicy is set to OnFailure, which means that the cronjob will automatically rerun the job if it fails, and also to trigger a rerun of the cronjob when the dan-test ConfigMap is updated, you can use the configmap.reloader.stakater.com/reload annotation on the cronjob

Something like this (but be careful if the cronjob is already running when the ConfigMap is updated, the job may be in progress and cannot be interrupted. In this scenario, you will need to delete the job manually to trigger a rerun) :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job-test
  annotations:
    configmap.reloader.stakater.com/reload: "dan-test"
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: job-test
            image: alpine
            command:
              - echo
              - hello $(DAN_TEST)
            env:
            - name: DAN_TEST
              valueFrom:
                configMapKeyRef:
                  name: dan-test
                  key: dan-test-1
          restartPolicy: OnFailure

good luck !