Kubernetes Ingress Controller using NLB with SSL passthrough in AWS gets TLS connection non-properly terminated

Question

I'm trying to set up kubernetes ingress controller in aws with ssl-passthrough. However when I curl -k -vvv https://<hostname> I get curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <hostname> When I try doing curl -k -vvv http://<hostname>, I get a 502 Bad Gateway.

The following is: A snippet of my k8s ingress controller service:

- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
    labels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 0.41.0
    name: ingress-controller-service
    namespace: ingress-controller
  spec:
    externalTrafficPolicy: Local
    loadBalancerSourceRanges:
    - 0.0.0.0/0
    ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    selector:
      app: ingress-controller
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
    type: LoadBalancer

Note that I've used the nlb annotations (as seen in the documentation)

The container arguments for my deployment:

        containers:
        - args:
          - /nginx-ingress-controller
          - --publish-service=$(POD_NAMESPACE)/ingress-controller-service
          - --election-id=ingress-controller-leader
          - --enable-ssl-passthrough
          - --configmap=$(POD_NAMESPACE)/ingress-controller-configmap
          - --ingress-class=test-shard-nginx
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --annotations-prefix=nginx.ingress.kubernetes.io

Note the --enable-ssl-passthrough argument

The container ports of my deployment:

          ports:
          - containerPort: 80
            name: http
            protocol: TCP
          - containerPort: 443
            name: https
            protocol: TCP

My ingress resource deployed in the namespace (with my ClusterIP type service that I want to route to) looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: test-shard-nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2020-11-25T05:49:40Z"
  generation: 1
  name: <some name>
  namespace: <some namespace>
  resourceVersion: "1257611736"
  selfLink: /apis/extensions/v1beta1/namespaces/<some namespace>/ingresses/<some name>
  uid: <uid>
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          serviceName: <service name>
          servicePort: 443
        path: /
status:
  loadBalancer:
    ingress:
    - hostname: <ingress controller service lb hostname>

Note the ssl-passthrough annotations

And here's the ClusterIP service I want my ingress controller to route to:

apiVersion: v1
kind: Service
metadata:
  annotations:
  creationTimestamp: "2020-11-25T05:49:02Z"
  labels:
    ttl: 3d
  name: <service name>
  namespace: <some namespace>
  resourceVersion: "1257608578"
  selfLink: /api/v1/namespaces/<some namespace>/services/<service name>
  uid: <uid>
spec:
  clusterIP: 10.3.157.207
  ports:
  - name: <port name>
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: <app name>
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

I've been stuck on this for a while. I was wondering what am I doing wrong? Thanks!