I am running a Spark driver on Kubernetes and it keeps failing with the error
WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
io.fabric8.kubernetes.client.WatcherException: too old resource version: 447057600 (447129816)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onStatus(AbstractWatchManager.java:395)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onMessage(AbstractWatchManager.java:363)
at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onMessage(WatcherWebSocketListener.java:52)
These are the versions I am using
Base Spark version 3.5.0
Hadoop version - 3.3.4
Spark operator version - v1beta2-1.3.8-3.1.1
The driver runs for a bit, throws the above message and goes into a szombie mode. I have gone through the spark logs, node description, pod statuses/description and all of them check out. I have also tried all solutions posted on Stack overflow and github before and none of them appear to work.
Anyone stuck on this message or fixed it before. Any help would be appreciated.