http app unreachable from tailscale and funnel and k8s

Question

I get an http app (written in rust) which serve 2 routes on port 3000:

1. /, for serving static file
2. /health, which return a simple string: "Ok"

When testing, everuthing works, both routes are accessibles from localhost.

I dockerise my app, expose port 3000, everythink work as expected.

I create a chart with helm create.

My deployment port look like :

ports:
  - name: http
    containerPort: 3000
    protocol: TCP

My service is

 spec:
  type: ClusterIP
  ports:
    - port: 443
      targetPort: http
      protocol: TCP
      name: http

I set 443 as port because of ingress.

Now I use an ingress with a vpn (Tailscale) to expose my service with funnel

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    tailscale.com/funnel: "true"
spec:
  ingressClassName: tailscale
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mysvc
            port:
              number: 443

All pods are running and I can see, into ingress description, the backend mysvc:443 with an ip.

I try to reach my http /health route with curl and tailnet name: curl -v subroute.tailnetname.ts.net:443/health.

But I get:

Host speedupdate.marlin-atlas.ts.net:443 was resolved.
* IPv6: 2a00:dd80:3a::131
* IPv4: 176.58.88.82
*   Trying 176.58.88.82:443...
* Connected to speedupdate.marlin-atlas.ts.net (176.58.88.82) port 443
> GET /health HTTP/1.1
> Host: speedupdate.marlin-atlas.ts.net:443
> User-Agent: curl/8.6.0
> Accept: */*
>
* Empty reply from server
* Closing connection

I don't inderstand why? Expected answer is the string Ok

I inspect proxy pod created by tailscale, and get no error log, no http call logs too.

So what is wrong ?

Update : some logs from pod proxy

2024/03/19 16:48:39 Drop: TCP{[fd7a:115c:a1e0:ab12:4843:cd96:625a:9516]:52274 > [fd7a:115c:a1e0::17df:370d]:60438} 80 no rules matched                                           

2024/03/19 16:48:39 Accept: TCP{[fd7a:115c:a1e0:ab12:4843:cd96:625a:9516]:52274 > [fd7a:115c:a1e0::17df:370d]:60438} 72 tcp non-syn

2024/03/19 16:48:39 Accept: TCP{[fd7a:115c:a1e0:ab12:4843:cd96:625a:9516]:52274 > [fd7a:115c:a1e0::17df:370d]:60438} 354 tcp non-syn

2024/03/19 16:48:40 http: proxy error: tls: first record does not look like a TLS handshake

Update 2: I can reach my app with kubectl port-forward mysvc

Answer 1

It looks like the problem you are having has to do with the way your Ingress is set up with the tailscale funnel.

As per this official doc on tailscale when you are exposing your kubernetes ingress resources to tailnet using tailscale kubernetes operator, these ingress resources only support TLS and can only be exposed over HTTPS.

To properly configure your tailscale deployment follow these steps mentioned in the official documentation:

Edit the Ingress resource you want to expose to use the Ingress class tailscale:

Set spec.ingressClassName to tailscale.

Set tls.hosts to the desired host name of the Tailscale node. Only the first label is used. See custom machine names for details.

For example, to expose an Ingress resource nginx to your tailnet:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  ingressClassName: tailscale
  tls:
  - hosts:
    - nginx

The backend is HTTP by default. To use HTTPS on the backend, either set the port name to https or the port number to 443:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        name: https
  ingressClassName: tailscale
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: https
    port: 443
    targetPort: 443
  type: ClusterIP

In the service definition which you have mentioned above instead of giving the port number you have mentioned http for the targetPort this might also be an issue. So, correct your service definition and the proper port number.

spec:
  type: ClusterIP
  ports:
    - port: 443
      targetPort: <port number>
      protocol: TCP
      name: http

Although the Ingress rule targets port 443, your service exposes port 3000 internally. Although you may be using a port for TLS termination with Tail Scale, communication between the Ingress and your service should take place on port 3000. Thus, change the backend service port number column for the Ingress rule to 3000.

This guarantees that traffic from the Ingress is routed to port 3000 on your service, where your application is listening. Verify again that the Tail Scale Funnel setup is configured to forward traffic on port 443(HTTPS) to the internal service port 3000 (HTTP).

Try utilizing curl localhost:3000/health to access the application directly from within the K8S cluster and kubectl exec into a pod to determine whether the problem is with your application or the Ingress settings.