how to set-up liveness and readiness probes for Celery worker pods

Question

I want to set-up liveness and readiness probes for Celery worker pods. Since these worker pods doesn't have a specific port associated to them I am finding it difficult. Main Django app nginx server was easier to set-up.

I am very new to k8s so not much familiar to the different ways to do it.

Answer 1

liveness probe for celery worker: This command only works when remote control is enabled.

$ celery inspect ping -d <worker_name> --timeout=<timeout_time>

When a celery worker uses a solo pool, healthcheck waits for the task to finish. In this case, you must increase the timeout waiting for a response.

so in yaml:

      livenessProbe:
        initialDelaySeconds: 45
        periodSeconds: 60
        timeoutSeconds: <timeout_time>
        exec:
          command:
          - "/bin/bash"
          - "-c"
          - "celery inspect ping -d <worker_name> | grep -q OK"

Of course you have to change the worker name and timeout to your own values

Answer 2

Basically, configuration files in YAML for Celery liveness and readiness probes can be set up in a general way as described in Kubernetes docs. You can specifically adjust them for Celery pods based on your requirements, for example:

        livenessProbe:
          exec:
            # bash is needed to replace the environment variable
            command: [
              "bash",
              "-c",
              "celery inspect ping -A apps -d celery@$HOSTNAME"
            ]
          initialDelaySeconds: 30
          periodSeconds: 60  
          timeoutSeconds: 10 

There is still ongoing GitHub issue on this matter - other solutions that might be useful for your set-up are described here.