Killing Sidecar container once main container is terminated in Jobs/Cron Jobs

Question

We are facing an issue wrt sidecars in Jobs/Cron Jobs. We are using EFK stack for logging and using filebeat as a sidecar container for shipping logs from app to ElasticSearch. But while implementing this in Batch Jobs, the sidecar container is not getting killed once the main container (main Job script) is Terminated . So the Job will never go to Completed/Terminated state. Any pointers on how to handle this issue. - To kill sidecar container once the main container is terminated.

Answer 1

From https://carlosbecker.com/posts/k8s-sidecar-shutdown/

I still need to test this out, but basically you have a shared volume between containers. The primary container has what's essentially a finally block with trap that touches a file when it exits.

Then the sidecar container polls for this file existing and exists when it does.

containers:
- name: agent
  command: ["/bin/sh", "-c"]
  args:
  - |
    trap 'touch /usr/share/pod/done' EXIT
    buildkite-agent-entrypoint start \
      --disconnect-after-job \
      --disconnect-after-idle-timeout=50    
# ...
  volumeMounts:
  - mountPath: /usr/share/pod
    name: tmp-pod
- name: dind
  command: ["/bin/sh", "-c"]
  args:
  - | 
    dockerd-entrypoint.sh &
    while ! test -f /usr/share/pod/done; do
      echo 'Waiting for the agent pod to finish...'
      sleep 5
    done
    echo "Agent pod finished, exiting"
    exit 0
# ...
  volumeMounts:
  - mountPath: /usr/share/pod
    name: tmp-pod
    readOnly: true
# ...
volumes:
- emptyDir: {}
  name: tmp-pod
# ...

Pretty nasty huh? Hopefully 1.28's first-class sidecar containers offer better support for this type of situation