fabric8 kubernetes client cannot connect to cluster: unexpected end of stream

Question

I am having trouble running a simple request (get pods) via fabric8 kubernetes client (latest version). The same get pods request works fine with kubectl get pods. Something is misconfigured but I cannot figure out what it is.

The setup: there is a tunnel from localhost to the kubernetes cluster API. The config is:

$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:6443
    tls-server-name: xxx.<redacted>NNNN.us-west-2.aws.host
  name: mycluster
contexts:
- context:
    cluster: mycluster
    namespace: default
    user: kube-aws-admin
  name: mycluster
current-context: mycluster
kind: Config
preferences: {}
users:
- name: kube-aws-admin
  user:
    token: REDACTED

I set the environment variable KUBECONFIG=$HOME/.kube/mycluster.conf and this works:

$ kubectl -n myns get pods
.... I get the correct output here ...

However, accessing the same cluster with the same KUBECONFIG via fabric8 kubernetes client does not work:

new KubernetesClientBuilder().build().pods().inNamespace("myns").list().getItems()....  and so on as usual

When I run this code, I get the error:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [myns]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:422)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:388)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:92)
    ...
Caused by: java.io.IOException: unexpected end of stream on Connection{127.0.0.1:6443, proxy=DIRECT hostAddress=/127.0.0.1:6443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=http/1.1}
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:515)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:420)
    ... 4 more
Caused by: java.io.IOException: unexpected end of stream on Connection{127.0.0.1:6443, proxy=DIRECT hostAddress=/127.0.0.1:6443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=http/1.1}
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:236)
    at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
    ... 19 more

Note that it tries to access 127.0.0.1:6443 which is correct, according to KUBECONFIG. It uses TLS, which is also correct.

Perhaps the problem is with tls-server-name that the fabric8 client does not understand?

I found that tls-server-name is parsed by fabric8 but does not seem to be used anywhere. If I remove this from the config, kubectl also stops working with an error Unable to connect to the server: EOF, which is similar to the "end of stream" error shown above.

I am using kubernetes 1.24, fabric8 6.7.2 and it works when this kind of code is run in a different situation where the access to the k8s cluster is configured without a tunnel.

I have tried:

• Setting KUBERNETES_MASTER=https://localhost:6443
• Setting NO_PROXY=localhost,127.0.0.1
• Adding code to set "context" to mycluster in the config for kubernetes client
• Setting some other options like KUBERNETES_DISABLE_AUTOCONFIG=true, KUBERNETES_BACKWARDSCOMPATIBILITYINTERCEPTOR_DISABLE=false, KUBERNETES_DISABLE_HOSTNAME_VERIFICATION=true, KUBERNETES_TRUST_CERTIFICATES=true, KUBERNETES_HTTP2_DISABLE=true, HTTP2_DISABLE=true
• Running kubectl proxy --port=8081 and KUBERNETES_MASTER=http://localhost:8081
• Using zulu JDK 8 and zulu JDK 17
• Using okHttp 4.11.0 and 5.0.0-alpha.11

None of the above helped. Sometimes I get Cause: java.io.IOException with message: unexpected end of stream on https://127.0.0.1:6443/... instead of a longer message shown above. But it is always failing with a similar message about end of stream.

• Is the problem that tls-server-name is being ignored by fabric8 kubernetes client?
• Is the problem with okhttp rather than with fabric8? I found that exactly the same error happens with the "official" java kubernetes client (https://github.com/kubernetes-client/java).

I looked at the source code of kubectl and kubernetes client-go but I could not find any code that used tls-server-name to connect to the kubernetes API in any kind of special way. So, while the kubernetes documentation says that tls-server-name is used for server certificate validation I could not find any code that does that.