Kubernetes check if pod fails

Question

In my Kubernetes based cluster I have few Java based, dockerized microservices, which are connected with each other, so every one sends and receives some information from each other. I have Kafka/Zookeeper cluster also which again connected to this microservices. I need to have some procedure like when I'm restarting active Kafka pod, another microservices should understand it and restart themselves. How to do this? Please give some example yml.

Answer 1

There is no Kubernetes native way to do your job. But you can make your application to do restart your container.

You can get help from containers' livenessProbe.

Kubernetes provides livenessProbe to detect where your application is live or not. liveness probes will actually attempt to restart a container if it fails.

spec:
  containers:
  - name: liveness
    image: <image>
    livenessProbe:
      httpGet: # make an HTTP request
        port: 8080 # port to use
        path: /live # endpoint to hit
        scheme: HTTP # or HTTPS
      initialDelaySeconds: 3
      periodSeconds: 3
      failureThreshold: 1

Now, make your application to return other than 200 when you need to restart.

If you can identify from your microservices that your kafka pods are restarted, then you return other than 200 as status code from /live http call.

If you want to call kafka service directly from liveness probe, add following under httpGet

httpGet:
  httpHeaders:
  - name: Host
    value: kafka-svc

Now, your application will call kafka-svc.svc:8080/live

Second way:

You can write an application which will watch your Kafka pods with kubernetes watch api, If this finds any restart in Kafka pods, then you can restart all dependent pods as you need.