Keda RabbitMQ - Keda not spawning additional jobs when queue has few messages

Question

I have a Keda Scaledjob configured to spawn 1 job per message having the state 'ready' in RabbitMQ.

It has a max replica count set to 70.

Observed:

• When there are many messages in the queue, let's say 300, Keda correctly creates new jobs to reach the max replica count limit => So there are 70 running jobs each consuming 1 message from the queue.

• When there are few messages in the queue, let's say 1 Ready and 1 Unacked, Keda refuses to create a new job even if there's enough resources in the cluster. It's like waiting until the current running job finishes to spawn a new job.

Here's my Keda configuration :

---
# Reference - https://keda.sh/docs/2.0/concepts/scaling-jobs/
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaledjob-puppeteer
  labels:
    environment: development
    app: puppeteer-display
spec:
  jobTargetRef:
    parallelism: 1                            # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    completions: 1                            # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
    activeDeadlineSeconds: 7200               # (2 hours) Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
    backoffLimit: 2                           # Specifies the number of retries before marking this job failed. Defaults to 6
    template:
      spec:
        volumes:
          ...
        containers:
          ...
  pollingInterval: 10
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 0
  maxReplicaCount: 75
  triggers:
    - type: rabbitmq
      metadata:
        protocol: amqp
        queueName: tasks
        mode: QueueLength
        value: "1"
      authenticationRef:
        name: keda-trigger-auth-rabbitmq-conn
---

How to make Keda to create a job whenever the queue has >= 1 message ?

Edit: It seems like it waits for at least 1 hour before creating the new job.

Answer 1

The problem seems to be the missing scalingStrategy setting. You can add following configuration:

  scalingStrategy:
    strategy: accurate

The accurate setting is used when you consume messages from your queue instead of locking the messages. This is often used in other message queues.

For reference you can look into https://keda.sh/docs/2.7/concepts/scaling-jobs/ You can find further information about the scaling strategies in the details section.