k8s/kind: Unable to connect to ingress controller via gRPC

Question

I'm on MacOS and using kind v0.10.0. I've followed the instructions given at https://kind.sigs.k8s.io/docs/user/ingress/ for setting up kind with an nginx ingress controller (and a private docker registry). Unfortunately, I am unable to connect to be ingress controller on 127.0.0.1:80. I've even tried moving my service, deployment, and ingress into the same namespace as the nginx ingress controller, i.e. namespace=ingress-nginx, but even that didn't fix anything.

Here's what I'm trying to do: route incoming gRPC messages to two separate backend services based on the gRPC method name.

Here's what I'm observing:

• If I open http://127.0.0.1 in my browser I see a 502 page served by nginx.

• I have even confirmed that the process listening to port 80 is com.docker.backend.

• However, when I try to connect to this port using a gRPC client, it just hangs/timesout:

  % ~/go/bin/grpcurl -plaintext localhost:80 list
  Failed to dial target host "localhost:80": context deadline exceeded
  

Can anyone help me debug this?

Pods

% kubectl get pods -n ingress-nginx                         
NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-b8tkt       0/1     Completed   0          13h
ingress-nginx-admission-patch-xddd7        0/1     Completed   2          13h
ingress-nginx-controller-9ddc777c5-bwnkx   1/1     Running     0          13h
route-guide-a-6fc997cbd-7zgbk              1/1     Running     0          10m
route-guide-a-6fc997cbd-hgpjv              1/1     Running     0          10m
route-guide-b-7b954b77bb-2p8vm             1/1     Running     0          10m
route-guide-b-7b954b77bb-nmdnt             1/1     Running     0          10m

Deployments

% kubectl get deployment -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           13h
route-guide-a              2/2     2            2           10m
route-guide-b              2/2     2            2           10m

Services

% kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.96.230.48    <none>        80:32236/TCP,443:32573/TCP   13h
ingress-nginx-controller-admission   ClusterIP   10.96.79.228    <none>        443/TCP                      13h
route-guide-a                        ClusterIP   10.96.249.171   <none>        10000/TCP                    11m
route-guide-b                        ClusterIP   10.96.143.194   <none>        10000/TCP                    11m

Ingress

% kubectl get ingress -n ingress-nginx
NAME                  CLASS    HOSTS   ADDRESS     PORTS   AGE
route-guide-ingress   <none>   *       localhost   80      12m

Config file

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: route-guide-a
  namespace: ingress-nginx
  labels:
    app: route-guide-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: route-guide-a
  template:
    metadata:
      labels:
        app: route-guide-a

    spec:
      containers:
        - name: route-guide-a
          image: localhost:5000/route_guide:v4
          imagePullPolicy: IfNotPresent
          command: ["/usr/local/go/bin/go", "run", "server/server.go"]
          ports:
            - containerPort: 10000
              name: grpc-port
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: route-guide-b
  namespace: ingress-nginx
  labels:
    app: route-guide-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: route-guide-b
  template:
    metadata:
      labels:
        app: route-guide-b

    spec:
      containers:
        - name: route-guide-b
          image: localhost:5000/route_guide:v4
          imagePullPolicy: IfNotPresent
          command: ["/usr/local/go/bin/go", "run", "server/server.go"]
          ports:
            - containerPort: 10000
              name: grpc-port
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

---
apiVersion: v1
kind: Service
metadata:
  name: route-guide-a
  namespace: ingress-nginx
  labels:
    app: route-guide-a
spec:
  # type: NodePort
  selector:
    app: route-guide-a
  ports:
    - port: 10000
      # nodePort: 31397
      targetPort: 10000
      name: grpc-port
      protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: route-guide-b
  namespace: ingress-nginx
  labels:
    app: route-guide-b
spec:
  # type: NodePort
  selector:
    app: route-guide-b
  ports:
    - port: 10000
      # nodePort: 31398
      targetPort: 10000
      name: grpc-port
      protocol: TCP

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: route-guide-ingress
  namespace: ingress-nginx
spec:
  rules:
  - http:
      paths: 
        - path: /routeguide.RouteGuide/GetFeature
          backend:
            serviceName: route-guide-a
            servicePort: grpc-port

        - path: /routeguide.RouteGuide/ListFeatures
          backend:
            serviceName: route-guide-a
            servicePort: grpc-port

        - path: /routeguide.RouteGuide/RecordRoute
          backend:
            serviceName: route-guide-b
            servicePort: grpc-port
        
        - path: /routeguide.RouteGuide/RouteChat
          backend:
            serviceName: route-guide-b
            servicePort: grpc-port

        # catch-all route
        - backend:
            serviceName: route-guide-b
            servicePort: grpc-port

Ingress config in the running cluster

Output of the following command: kubectl get ingress -n ingress-nginx route-guide-ingress -o yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/backend-protocol":"GRPC"},"name":"route-guide-ingress","namespace":"ingress-nginx"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"route-guide-a","servicePort":"grpc-port"},"path":"/routeguide.RouteGuide/GetFeature"},{"backend":{"serviceName":"route-guide-a","servicePort":"grpc-port"},"path":"/routeguide.RouteGuide/ListFeatures"},{"backend":{"serviceName":"route-guide-b","servicePort":"grpc-port"},"path":"/routeguide.RouteGuide/RecordRoute"},{"backend":{"serviceName":"route-guide-b","servicePort":"grpc-port"},"path":"/routeguide.RouteGuide/RouteChat"},{"backend":{"serviceName":"route-guide-a","servicePort":"grpc-port"}}]}}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
  creationTimestamp: "2021-05-02T02:57:24Z"
  generation: 2
  managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:kubernetes.io/ingress.class: {}
          f:nginx.ingress.kubernetes.io/backend-protocol: {}
      f:spec:
        f:rules: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-05-02T02:57:24Z"
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: nginx-ingress-controller
    operation: Update
    time: "2021-05-02T02:57:44Z"
  name: route-guide-ingress
  namespace: ingress-nginx
  resourceVersion: "12115"
  uid: be3862b4-13a7-4b96-b5f8-399e28bb373c
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/GetFeature
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/ListFeatures
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-b
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/RecordRoute
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-b
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/RouteChat
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: localhost

Answer 1

According to kubernetes/ingress-nginx#2444 and kubernetes/ingress-nginx#6313, Nginx cannot multiplex HTTP/1.1 and HTTP/2 in Port 80. As shown below, http2 is not enabled in Port 80 by default.

$ kubectl exec -it -n $INGRESS_CONTROLLER_NS $INGRESS_CONTROLLER_POD -- cat /etc/nginx/nginx.conf
...
server {
    server_name your.domain ;

    listen 80  ;
    listen [::]:80  ;
    listen 443  ssl http2 ;
    listen [::]:443  ssl http2 ;
}
...

As a result, some probable solutions could be:

• Use SSL instead.
• Use a custom template as mentioned in the comment.