AWS EKS Load Balancer Controller: Ingress: gRPC Configuration

Question

I am deploying one microservice in AWS EKS which is intended to communicate with our external client over gRPC.

Brief Description:

I have one backend deployment that runs on internal port 5051 and also on external container port 443. I have aligned service for the same which listens on backend pod's port 5051 and exposes the same port as NodePort. We have an Ingress controller which holds the backend service as gRPC and forwards traffic on its endpoint to the backend service and Listens on HTTPS 443 Port with proper certificates (IAM Certificates) and Domain.

I can telnet the domain on both ports:

But seems the gRPC communication is not working. (Getting Operation Cancelled on Postman and also socket error from code).

I am very new to configuring gRPC in AWS EKS using AWS Load Balancer Controller and can't properly debug the whole setup.

I have followed this blog post for my configuration and have added things as per my previous experience in configuring Ingress in AWS EKS.

Here are the manifestes:

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my_app
  namespace: ignite
spec:
  selector:
    matchLabels:
      run: my_app
  replicas: 1
  template:
    metadata:
      labels:
        run: my_app
    spec:
      terminationGracePeriodSeconds: 300
      containers:
      - name: my_app
        image: <Image Tag>
        imagePullPolicy: Always
        ports:
        - containerPort: 5051
        resources:
          limits:
            cpu: 2000m
            memory: 4096Mi
          requests:
            cpu: 1000m
            memory: 2048Mi
        envFrom:
        - configMapRef:
            name: my_app-configmap
        volumeMounts:
          - name: persistent-storage
            mountPath: /Audio_data
          - name: my_app-config
            mountPath: "/cfg-1/app/config.json"
            subPath: config.json
            readOnly: true
          - name: my_app-config-1
            mountPath: "/cfg-2/app/config.json"
            subPath: config.json
            readOnly: true
      volumes:
        - name: my_app-config
          secret:
            secretName: my_app-ex-secret
        - name: my_app-config-1
          secret:
            secretName: my_app-ex-secret-1
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: audio-data

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: my_app
  namespace: ignite
  labels:
    run: my_app
spec:
  selector:
    run: my_app
  ports:
  - port: 5051
    protocol: TCP
    targetPort: 5051
  externalTrafficPolicy: Local
  type: NodePort

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: ignite
  name: my_app
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: subnet-1,subnet-2
    alb.ingress.kubernetes.io/certificate-arn: <Cert ARN>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/connection-idle-timeout: "7200"
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=4000
    alb.ingress.kubernetes.io/backend-protocol-version: GRPC
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
    alb.ingress.kubernetes.io/success-codes: '12'
spec:
  ingressClassName: alb
  rules:
  - host: my.domain.com
    http:
      paths:
      - backend:
          service:
            name: my_app
            port:
              number: 5051
        path: /
        pathType: Prefix