Keda Operator Access Denied for SQS

Question

I am using SQS scaler for Keda to scale a deployment. At present I am using operator based identityOwner for scaledObject. I have build an IAM role with following permissions and trust relationship :-

permission

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "KedaSQSReadPermission",
            "Effect": "Allow",
            "Action": "sqs:GetQueueAttributes",
            "Resource": "arn:aws:sqs:<region-name>:<account-id>:<test-queue>"
        },
        {
            "Sid": "KedaSQSListPermission",
            "Effect": "Allow",
            "Action": "sqs:ListQueues",
            "Resource": ["*"]
        }
    ]
}

trust relationship

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>:sub": "system:serviceaccount:keda:keda-operator"
                }
            }
        }
    ]
}

This role is annotated in keda-operator service account in keda namespace as detailed below :-

{
  "apiVersion": "v1",
  "kind": "ServiceAccount",
  "metadata": {
    "annotations": {
      "eks.amazonaws.com/role-arn": "arn:aws:iam::<account-id>:role/<role-name>",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/name\":\"keda-operator\",\"app.kubernetes.io/part-of\":\"keda-operator\",\"app.kubernetes.io/version\":\"2.7.0\"},\"name\":\"keda-operator\",\"namespace\":\"keda\"}}\n"
    },
    "creationTimestamp": "2022-07-14T13:49:40Z",
    "labels": {
      "app.kubernetes.io/name": "keda-operator",
      "app.kubernetes.io/part-of": "keda-operator",
      "app.kubernetes.io/version": "2.7.0"
    },
    "name": "keda-operator",
    "namespace": "keda",
    "resourceVersion": "174401872",
    "uid": "a5862afb-fcd1-49e3-8ebb-6c81f4fd5014"
  },
  "secrets": [
    {
      "name": "keda-operator-token-wzqsz"
    }
  ]
}

I have a deployment running in different namespace - test-keda. For scaling against queue length, I using following spec :-

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
    name: scaled-object-test
    namespace: test-keda
    labels:
        name: test-app # Required Name of the deployment we want to scale.
spec:
  scaleTargetRef:
    kind: Deployment
    name: test-app
  pollingInterval: 5
  cooldownPeriod: 10
  maxReplicaCount: 8
  minReplicaCount: 0
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.<region-name>.amazonaws.com/<account-id>/<queue-name>
        queueLength: "10"
        awsRegion: "<region-name>"
        identityOwner: operator

I have used official docs for SQS scaler for setting up above configurations. However, upon checking keda-opeator pod log, I find following error :-

error": "AccessDenied: Access to the resource https://sqs.<region-name>.amazonaws.com/ is denied.\n\tstatu │
│ s code: 403, request id: 87fd482d-89cf-54ef-af3e-13a25f4d9e23"

I tried using pod as identityOwner for my scaledobject, but seems to throw Access Denied, due to node-role trying to assume deployment IRSA role.

Am I missing something here ? Would appreciate any hint on where am I going wrong.

Thanks

Answer 1

Restarting keda deployments solved the issue. Run following commands :-

kubectl rollout restart deployment keda-metrics-apiserver -n keda
kubectl rollout restart deployment keda-operator -n keda