HEALTH CHECK not resolving for ClamAV hosted in GKE cluster

Question

Couldn't resolve health check related issue for the clamAV:1.2 deployment hosted in the GKE cluster in the port 3310, 7357.

I'm new to this clamAV concept and hosting in GKE with ingress

Facing some backend services are in UNHEALTHY state, in the ingress after adding routing path to the service defined.

I've deployed a clamAV:1.2 docker image in the GKE cluster through deployment file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: clam-av
spec:
  replicas: 1
  selector:
    matchLabels:
      run: clam-av
  template:
    metadata:
      labels:
        run: clam-av
    spec:
      nodeSelector:
        cloud.google.com/gke-nodepool: XXX-XXX-pool
      terminationGracePeriodSeconds: 60
      containers:
      - name: clamav-container
        image: clamav/clamav:1.2
        resources:
          requests:
            cpu: 200m
            memory: 1Gi
        imagePullPolicy: Always       
        ports:
        - containerPort: 3310
        # - containerPort: 7357
       

Created a service file for this deployment:

apiVersion: v1
kind: Service
metadata:
  name: clam-av-service
  annotations:
    cloud.google.com/backend-config: '{"default": "backend-for-clamAV"}'
spec:
  selector:
    run: clam-av
  ports:
  - name: http3310
    protocol: TCP
    port: 80
    targetPort: 3310
  # - name: http7357
  #   protocol: TCP
  #   port: 80
  #   targetPort: 7357
  type: ClusterIP

Also created a BackendConfig:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: backend-for-clamAV
spec:
  timeoutSec: 150
  connectionDraining:
    drainingTimeoutSec: 150
  healthCheck:
    checkIntervalSec: 15
    port: 80
    type: HTTP
    requestPath: /
    healthyThreshold: 1
    unhealthyThreshold: 3
    timeoutSec: 15

Tried setting the spec.healthcheck.type as TCP as the clamAV 3310 is a TCP connection. But the GCP ingress doesn't supports TCP.

Any suggestion to resolve this issue would be appreciated. Thank You !

Answer 1

The error message is Some backend services are in UNHEALTHY state indicating that only a few of your backends are affected and not all backend services are in UNHEALTHY state.

Try below troubleshooting steps to resolve the issue

1. Check if all your pods are running, that all containers within pods are ready and pod status is Running. Using the below command.

   $ kubectl get po -n <namespace>

   Check logs of suspicious pods by using below command:

   $ kubectl logs <podname> -c <containerName>

   In general you should check all pods pointing to the load balancer.

2. Confirm if livenessProbe and readinessProbe are configured properly and response is 200.

   As mentioned in the documentation:

Currently, all service backends must satisfy either of the following requirements to pass the HTTP health checks sent to it from the GCE load balancer: 1. Respond with a 200 on '/'. The content does not matter. 2. Expose an arbitrary url as a readiness probe on the pods backing the Service.

Make sure that the readinessProbe is pointing to the same port that you expose to the Ingress.

3. Describe your ingress $ kubectl describe ingress <yourIngressName> and check backend.

4. Check if service is correctly listening on port 80 by running command netstat -tnl \| grep 80.

5. Check health check logs for the backend service which will return a response code which helps to debug further.