Liveness probe failed: Get-http context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Question

After a DO update to 1.24.16-do.0 , my cloud-config pod encountered Liveness probe failed: Get-<http.ip> context deadline exceeded (Client.Timeout exceeded while awaiting headers). When i curl the ip, i'm getting a 137 error, as the pod is backing off i suppose. The traffic is very less and the memory/cpu/threads is much beyond limits thresholds. The issue is reproduced on different cluster compute nodes. Any other resources were also not changed during the update. deployment of my liveness-probe

http-get http://:8888/actuator/health delay=180s 

the image is being pulled by an internal registry and works, too. I also tried to disable all the components that being checked as part of the actuator health check, but nothing has changed. liveness config:

livenessProbe:
      failureThreshold: 4
      httpGet:
        path: /actuator/health
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 180
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 5

i'd be gratefule for any hint

Answer 1

You encountered the error: "Client.Timeout exceeded while awaiting headers", which means that the probe was considered to be failed by the Kubernetes as it didn't responded in specified time.

You need to increase your timeoutSeconds to 10s for livenessProbe.

timeoutSeconds: This parameter is part of the configuration for both liveness and readiness probes. It specifies the number of seconds after which the probe times out. The default value is 1 second. If a probe doesn’t respond within the specified timeoutSeconds, Kubernetes considers the probe to have failed.