S3 - Kubernetes probe

Question

I have the following situation:

Application uses S3 to store data in Amazon. Application is deployed as a pod in kubernetes. Sometimes some of developers messes with access data for S3 (eg. user/password) and application fails to connect to S3 - but pod starts normally and kills previous pod version that worked OK (since all readiness and aliveness probes are OK). I thought of adding S3 probe to readiness - in order to execute HeadBucketRequest on S3 and if this one succeeds it is able to connect to S3. The problem here is that these requests cost money, and I really need them only on start of the pod.

Are there any best-practices related to this one?

Answer 1

If you (quote) "... really need them [the probes] only on start of the pod" then look into adding a startup probe.

In addition to what startup probes help with - pods that take longer time to start - a startup probe will make it possible to verify a condition only at pod startup time.

Answer 2

Readiness and liveness prove as for checking the health of POD or container while running. You scenario is quite wired but with Readiness & liveness probe it wont work as it fire on internal and which cost money.

in this case you might can use the lifecycle hook :

containers:
      - image: MAGE_NAME
        lifecycle:
          postStart:
            exec:
              command:  ["/bin/sh", "-c", "script.sh"]

which will run the hook at starting of the container you can keep shell file inside the POD or image.

inside shell file you can right logic if 200 response move a head and container get started.

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/