Resolving AssumeRoleWithWebIdentity

Question

Following this guide, I cannot resolve the final fifth step, which fails with:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
command terminated with exit code 254

My ultimate goal is to have a boto3 client in a pod in a particular namespace perform EC2 actions. I am thinking my role, RBAC, or arn's I am using are not correct. My manifest is at bottom.

Edit: I am getting the AWS envs within the pod, see image.

My role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account_number>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/<oidc_number>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/<oidc_number>:sub": "system:serviceaccount:reporting-analytics-services-kafka-tools:",
                    "oidc.eks.us-east-1.amazonaws.com/id/<oidc_number>:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

My policy (which is attached to the role, I know it's overkill right now):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:*",
                "sts:*",
                "ec2:describeInstances",
                "ec2:stopInstances",
                "ec2:startInstances",
                "iam:ListRoles"
            ],
            "Resource": "*"
        }
    ]
}

This is what my deployment file looks like.

---
# Source: kafka-tools-pod/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: test-pod
  name: test-pod
  namespace: reporting-analytics-services-kafka-tools
spec:
  ports:
    - port: 8080
  selector:
    app: test-pod
---
# Source: kafka-tools-pod/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-pod
  name: test-pod
  namespace: reporting-analytics-services-kafka-tools
spec:
  replicas: 3
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: test-pod
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: test-pod
      annotations:
        Error: "invalid character '$' looking for beginning of value"
    spec:
      serviceAccountName: kafka-tools-pod-service-account
      containers:
        - envFrom:
            - configMapRef:
                name: kafka-tools-pod-env
          image: <somenumber>.dkr.ecr.us-east-1.amazonaws.com/kafka-testing:latest
          imagePullPolicy: IfNotPresent
          name: test-pod
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 1500m
              memory: 3Gi
            requests:
              cpu: 350m
              memory: 128Mi
---
# Source: kafka-tools-pod/templates/ingress-internal.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: internal-nginx
  name: kafka-tools-pod-internal
  namespace: reporting-analytics-services-kafka-tools
spec:
  rules:
    - host: mycompany.use1.develop.percipio.aws
      http:
        paths:
          - backend:
              service:
                name: test-pod
                port:
                  number: 8080
            path: /
            pathType: Prefix
---
# Source: kafka-tools-pod/templates/verticalpodautoscaler.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  labels:
    app: test-pod
  name: test-pod
  namespace: reporting-analytics-services-kafka-tools
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test-pod
  updatePolicy:
    updateMode: 'Off'
--- 
apiVersion: v1 
kind: ServiceAccount 
metadata:
  name: kafka-tools-pod-service-account
  namespace: reporting-analytics-services-kafka-tools
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<myaccount>:role/kafka-nodes-reboot-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kakfa-tools-pod-role
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["extensions"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["apps"]
  resources: ["*"]
  verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kakfa-tools-pod-role-binding
subjects:
- kind: ServiceAccount
  name: kafka-tools-pod-service-account
  namespace: reporting-analytics-services-kafka-tools
roleRef:
  kind: Role
  name: kakfa-tools-pod-role
  apiGroup: rbac.authorization.k8s.io

Also looked at answers here, here and used the official guide here.

Answer 1

A few things helped solve the issue.

1. Add "sts:AssumeRole" to the policy.
2. As @jordanm suggests edit the arn to the exact pod.
3. Role and roleBinding were configured incorrectly:

--- 
apiVersion: v1 
kind: ServiceAccount 
metadata:
  name: kafka-tools-pod-service-account
  namespace: reporting-analytics-services-kafka-tools
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<accountnumber>:role/KafkaRebootRole
    eks.amazonaws.com/audience: "sts.amazonaws.com"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kafka-role
  namespace: reporting-analytics-services-kafka-tools
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kafka-rolebinding
  namespace: reporting-analytics-services-kafka-tools 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kafka-role
subjects:
- namespace: reporting-analytics-services-kafka-tools 
  kind: ServiceAccount
  name: kafka-tools-pod-service-account

After this, running aws sts get-caller-identity in the pod or using a boto3 client does everything I need it to.