Continuous TLS handshake error logs in vault nodes due to LB health check

Question

I am getting continuous TLS handshake errors every 5 sec due to my load balancer pinging vault nodes in every 5 seconds. Kube load balancer is pinging my vault nodes using

nc -vz podip podPort every 5 sec

I have already disabled client cert verification in my config.hcl but still see below logs in my kubectl logs for vault

kubectl logs pod-0 -n mynamespace 
[INFO]  http: TLS handshake error from 10.x.x.x:60056: EOF 2020-09-02T01:13:32.957Z 
[INFO]  http: TLS handshake error from 10.x.x.x:23995: EOF 2020-09-02T01:13:37.957Z 
[INFO]  http: TLS handshake error from 10.x.x.x:54165: EOF 2020-09-02T01:13:42.957Z 

Below is my config.hcl which I am loading via kube config map

apiVersion: v1
kind: ConfigMap
metadata:
  name: raft-config
  labels:
    name: raft-config
data:
  config.hcl: |
    storage "raft" {
      path  = "/vault-data"
      tls_skip_verify = "true"
      retry_join {
        leader_api_addr = "https://vault-cluster-0:8200"
        leader_ca_cert_file = "/opt/ca/vault.crt"
        leader_client_cert_file = "/opt/ca/vault.crt"
        leader_client_key_file = "/opt/ca/vault.key"
      }
      retry_join {
        leader_api_addr = "https://vault-cluster-1:8200"
        leader_ca_cert_file = "/opt/ca/vault.crt"
        leader_client_cert_file = "/opt/ca/vault.crt"
        leader_client_key_file = "/opt/ca/vault.key"
      }
      retry_join {
        leader_api_addr = "https://vault-cluster-2:8200"
        leader_ca_cert_file = "/opt/ca/vault.crt"
        leader_client_cert_file = "/opt/ca/vault.crt"
        leader_client_key_file = "/opt/ca/vault.key"
      }
    }
    seal "transit" {
     address = "https://vaulttransit:8200"
     disable_renewal = "false"
     key_name = "autounseal"
     mount_path = "transit/"
     tls_skip_verify = "true"
    }
    listener "tcp" {
     address = "0.0.0.0:8200"
     tls_cert_file = "/opt/ca/vault.crt"
     tls_key_file = "/opt/ca/vault.key"
     tls_skip_verify = "true"
     tls_disable_client_certs = "true"
    }
    ui=true
    disable_mlock = true

As I am using external open source vault image and my load balancer is an internal LB (which has internal CA cert). I am suspecting my vault pod is not able to recognize the CA cert provided by my load balancer when it tries to ping port 8200(TCP listener is started by vault on this port)

These logs are harmless and not causing any issue but they are unnecessary noise which I want to avoid. My vault nodes are working on https and there seems to be no issue in their functionality.

Can someone please help understand why vault TCP listener is trying to do TLS handshake even though I have explicitly specified tls_disable_client_certs = "true"

Again these logs are flooding my pods every 5 sec when my LB tries to do a health check on my pods using nc -vz podip podPort

My vault version is 1.5.3

Answer 1

The messages are not about client certs or CA certs, a TLS handshake happens whether the client presents a certificate or not.

Instead, it is because a TCP connection is created and established and the Go library now wants to start a TLS handshake. Instead, the other side (the health checker) just hangs up and the TLS handshake never happens. Go then logs this message.

You are correct in saying that it is harmless, this is purely a side effect of port-liveness health checking. It is however spammy and annoying.

You have two basic options to get around this:

1. filter the messages out of the logs when persisting them
2. change to a different type of health check

I would recommend the second option: switch to a different health check. Vault has a /sys/health endpoint that can be used with HTTPS health checks.

In addition to getting rid of the TLS warning messages, the health endpoint also allows to you check for active and unsealed nodes.