How to set maxReplicas based on queue size for HorizontalPodAutoscaler?

728 Views Asked by At

In our Kubernetes cluster I have an HPA configured with external metrics to monitor a queue size. Furthermore I have the HPAScaleToZero gate enabled so that if there are no messages on the queue there will be no pods running. Most of the time this works fine but occasionally large numbers of messages are pushed onto the queue. When this happens I need more than 1 pod to clear the backlog. I haven't been able to figure out how to pin the number of pods to the size of the queue without HPA spinning up more than I need if just 1 message is pushed to the queue. I would like it to scale like the following with a hard max of 3 pods.

0 messages -> 0 pods
>=1 message -> 1 pod
>=10000 messages -> 2 pods
>=20000 messages -> 3 pods

I tried creating multiple HPA definitions but they ended up conflicting and battling over whose "maxReplicas" to use. The SinglePod spec would create 1 pod. Then the MultiPod would create 2 new ones and kill off the one from SinglePod. Then SinglePod would kick back in and kill the 2 from MultiPod. This went back and forth until the queue size fell below the MultiPod threshold.

SinglePod Spec:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
spec:
  maxReplicas: 1
  minReplicas: 0
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: jms-queue-processor
  metrics:
  - type: External
    external:
      metric:
        name: flux-query
        selector:
          matchLabels:
            query-name: process-queue-size
      target:
        type: Value
        value: "1"

MultiPod Spec:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
spec:
  maxReplicas: 3
  minReplicas: 0
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: jms-queue-processor
  metrics:
  - type: External
    external:
      metric:
        name: flux-query
        selector:
          matchLabels:
            query-name: process-queue-size
      target:
        type: Value
        value: "10000"

Any suggestions on how to accomplish this? Thanks, Steve

0

There are 0 best solutions below