aws camunda zeebe ClientStatusException: deadline exceeded after 19.999991631s

Question

I am getting the below exception in my application while processing the messages using camunda zeebe (using aws eks).
While sequential message insertion on a single thread is working, parallel insertion (multithreaded) is not.
It seems like the zeebe worker is not able to handle the load with my existing zeebe cluster configuration.

I tried multiple zeebe configurations like 6/12 broker, 1/24 partitions but getting same issue.

Exception-

io.camunda.zeebe.client.api.command.ClientStatusException: deadline exceeded after 19.999995604s. [remote_addr=a6015d115a-128.us-east-1.elb.amazonaws.com/5.2.2.1:8080]
            at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:93)
            at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:50)
            at com.db.matchservice.service.WorkFlowAction.executeWorkflow(WorkFlowAction.java:63)

Caused by: java.util.concurrent.ExecutionException:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19.999995604s. [remote_addr=a6015d115a-128.us-east-1.elb.amazonaws.com/5.2.2.1:8080]
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)

AWS Instance details-

InstanceType-c5.12xlarge, IOPS-16000, SSD-250, CPU-8, RAM-32

Zeebe cluster configuration:

spec:
      volumes:
        - name: config
          configMap:
            name: zeebe-cluster-zeebe-cluster-helm
            defaultMode: 484
        - name: exporters
          emptyDir: {}
      containers:
        - name: zeebe-cluster-helm
          image: 'camunda/zeebe:1.1.2'
          ports:
            - name: http
              containerPort: 9600
              protocol: TCP
            - name: command
              containerPort: 26501
              protocol: TCP
            - name: internal
              containerPort: 26502
              protocol: TCP
          env:
            - name: ZEEBE_BROKER_CLUSTER_CLUSTERNAME
              value: zeebe-cluster-zeebe
            - name: ZEEBE_LOG_LEVEL
              value: info
            - name: ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT
              value: '24'
            - name: ZEEBE_BROKER_CLUSTER_CLUSTERSIZE
              value: '6'
            - name: ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR
              value: '1'
            - name: ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
              value: '24'
            - name: ZEEBE_BROKER_THREADS_IOTHREADCOUNT
              value: '24'
            - name: ZEEBE_BROKER_GATEWAY_ENABLE
              value: 'false'
            - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME
              value: io.camunda.zeebe.exporter.ElasticsearchExporter
            - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL
              value: 'http://elasticsearch-master:9200'
            - name: ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT
              value: '26501'
            - name: ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT
              value: '26502'
            - name: ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT
              value: '9600'
            - name: K8S_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError
                -XX:HeapDumpPath=/usr/local/zeebe/data
                -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log
                -XX:+ExitOnOutOfMemoryError
          resources:
            limits:
              cpu: '1'
              memory: 4Gi
            requests:
              cpu: 500m
              memory: 2Gi
          volumeMounts:
            - name: config
              mountPath: /usr/local/zeebe/config/application.yaml
              subPath: application.yaml
            - name: config
              mountPath: /usr/local/bin/startup.sh
              subPath: startup.sh
            - name: data
              mountPath: /usr/local/zeebe/data
            - name: exporters
              mountPath: /exporters
          readinessProbe:
            httpGet:
              path: /ready
              port: 9600
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        volumeMode: Filesystem
      status:
        phase: Pending
  serviceName: zeebe-cluster-zeebe
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  revisionHistoryLimit: 10
status:
  observedGeneration: 17
  replicas: 6
  readyReplicas: 6
  currentReplicas: 6
  updatedReplicas: 6
  currentRevision: zeebe-cluster-zeebe-55c88fdf8f
  updateRevision: zeebe-cluster-zeebe-55c88fdf8f
  collisionCount: 0