How to setup an authenticated HTTP check to Kubernetes API with HAProxy

Question

I have HAProxy load balancing my Kubernetes cluster using TCP health checks as described by k3s documentation. I found that my nodes sometimes become "Unready", but the nodes still reply to the TCP checks, making requests to the cluster unreliable.

To solve this issue, I want to use the Kubernetes health endpoints (/livez and /readyz), but the api is authenticated. I have created a service account with a long lived token and I have bound the monitoring role to it. Now I am able to curl the api point with the following command curl https://192.168.1.201:6443/healthz --cacert /usr/local/share/ca-certificates/ca.crt --header "Authorization: Bearer $TOKEN". However, I cannot for the life of me create a HAProxy config that forwards all traffic without touching it, and does an authenticated health check to the Kubernetes API at the same time. Any suggestions on a HAProxy configuration would be much appreciated.

My current HAProxy setup:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

defaults
        log     global
        mode    tcp
        option  tcplog
        option logasap
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend k3s-frontend-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend-api

backend k3s-backend-api
    mode tcp
    option tcp-check

    balance leastconn
    default-server downinter 5s
    server server-1 192.168.1.201:6443 check
    server server-2 192.168.1.202:6443 check
    server server-3 192.168.1.203:6443 check

frontend k3s-frontend-http
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend-http

backend k3s-backend-http
    mode tcp
    option tcp-check

    balance leastconn
    default-server downinter 5s
    server server-1 192.168.1.201:80 check
    server server-2 192.168.1.202:80 check
    server server-3 192.168.1.203:80 check

frontend k3s-frontend-https
    bind *:443
    mode tcp
    option tcplog
    default_backend k3s-backend-https

backend k3s-backend-https
    mode tcp
    option tcp-check

    balance leastconn
    default-server downinter 5s
    server server-1 192.168.1.201:443 check
    server server-2 192.168.1.202:443 check
    server server-3 192.168.1.203:443 check

frontend stats
    mode http
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth user:pass

Answer 1

You can create HTTP check for backend in TCP mode, but your config has only option tcp-check. Instead go for HTTP check:

option httpchk
http-check send meth GET uri /healthz hdr authorization "Bearer $TOKEN"

This makes HTTP/1.0 check. If it doesn't work, then do HTTP/1.1 by adding ver HTTP/1.1 hdr host <hostname_of_your_service> after uri. http-check connect ssl may also be needed, because your servers don't mention ssl, because it's TCP mode.

Answer 2

It ended up being a combination of a typo (k3s-frontend-http said 6443 instead of 80), a service on my cluster gone rogue and a missing haproxy command. The working file I ended up with looks as follows:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

defaults
        log     global
        mode    tcp
        option  tcplog
        option  logasap
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend k3s-frontend-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend-api

backend k3s-backend-api
    mode tcp
    option httpchk
    http-check connect port 6443 ssl
    http-check send meth GET uri /livez hdr Authorization "Bearer {YOUR_TOKEN}"

    balance leastconn
    default-server downinter 5s check ca-file /usr/local/share/ca-certificates/ca.crt
    server server-1 192.168.1.201:6443  
    server server-2 192.168.1.202:6443  
    server server-3 192.168.1.203:6443 

frontend k3s-frontend-http
    bind *:80
    mode tcp
    option tcplog
    default_backend k3s-backend-http

backend k3s-backend-http
    mode tcp
    option httpchk
    http-check connect port 6443 ssl
    http-check send meth GET uri /livez hdr Authorization "Bearer {YOUR_TOKEN}"

    balance leastconn
    default-server downinter 5s check ca-file /usr/local/share/ca-certificates/ca.crt
    server server-1 192.168.1.201:80 
    server server-2 192.168.1.202:80 
    server server-3 192.168.1.203:80 

frontend k3s-frontend-https
    bind *:443
    mode tcp
    option tcplog
    default_backend k3s-backend-https

backend k3s-backend-https
    mode tcp
    option httpchk
    http-check connect port 6443 ssl
    http-check send meth GET uri /livez hdr Authorization "Bearer {YOUR_TOKEN}"

    balance leastconn
    default-server downinter 5s check ca-file /usr/local/share/ca-certificates/ca.crt
    server server-1 192.168.1.201:443 
    server server-2 192.168.1.202:443 
    server server-3 192.168.1.203:443 

frontend stats
    mode http
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth {YOUR_USER}:{YOUR_PASS}