Quarkus Kubernetes Client - TimeoutException

Question

We have upgraded Quarkus from 2.11.1 to 3.6.6. In earlier version 2.11.1. Kubernetes Client worked smoothly, but on version 3.6.6 Kubernetes client gave Timeout exception.

Exception details - (vert.x-worker-thread-10) Exception thrown during asynchronous load: io.fabric8.kubernetes.client.KubernetesClientException.

When we add @Blocking(io.smallrye.common.annotation) Annotation on method then it works. But here making this blocking behavior is not a good practice.

Complete exception.

(vert.x-worker-thread-10) Exception thrown during asynchronous load: io.fabric8.kubernetes.client.KubernetesClientException
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:791)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:192)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:148)
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:97)
        at com.rockwell.idp.security.identity.cache.IdentityProviderCache.getConfigMap(IdentityProviderCache.java:112)
        at com.rockwell.idp.security.identity.cache.IdentityProviderCache.lambda$getIdentityProviderDetailsByTenantId$0(IdentityProviderCache.java:103)
        at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
        at io.smallrye.mutiny.vertx.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:35)
        at io.smallrye.mutiny.vertx.DelegatingHandler.handle(DelegatingHandler.java:25)
        at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
        at io.vertx.core.impl.WorkerContext.execute(WorkerContext.java:111)
        at io.vertx.core.impl.WorkerContext.execute(WorkerContext.java:53)
        at io.vertx.core.impl.WorkerContext.execute(WorkerContext.java:65)
        at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:184)
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51)
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
        at io.vertx.core.Promise.complete(Promise.java:66)
        at io.vertx.core.eventbus.impl.ReplyHandler.dispatch(ReplyHandler.java:97)
        at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.execute(HandlerRegistration.java:137)
        at io.vertx.core.eventbus.impl.DeliveryContextBase.next(DeliveryContextBase.java:80)
        at io.vertx.core.eventbus.impl.DeliveryContextBase.dispatch(DeliveryContextBase.java:43)
        at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:98)
        at io.vertx.core.eventbus.impl.ReplyHandler.doReceive(ReplyHandler.java:81)
        at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:49)
        at io.vertx.core.impl.WorkerContext.lambda$null$1(WorkerContext.java:92)
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.TimeoutException
        at io.fabric8.kubernetes.client.utils.AsyncUtils.lambda$withTimeout$0(AsyncUtils.java:42)
        at io.fabric8.kubernetes.client.utils.Utils.lambda$schedule$6(Utils.java:473)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        ... 1 more

Expecting to work without using @Blocking annotation

Answer 1

When we add @Blocking (io.smallrye.common.annotation) Annotation on method then it works.

Yes, this is the workaround that will prevent these types of issues from popping up in the more strict Quarkus 3.x version (regarding Vert.x reactive code).

But here making this blocking behavior is not a good practice.

It would be good to see the code that triggers the exception to understand better.

Currently, all of the KubernetesClient calls are blocking. The client performs calls to the Kube API server in a synchronous fashion. You need to either use the @Blocking annotation to make the complete logic of that method blocking or move the KubernetesClient related logic somewhere else so that it's processed asynchronously.

You can check the Repo's issue for more information https://github.com/quarkusio/quarkus/issues/38133