Java Version for server and agent
openjdk 17.0.9 2023-10-17
OpenJDK Runtime Environment (build 17.0.9+9-Ubuntu-122.04)
OpenJDK 64-Bit Server VM (build 17.0.9+9-Ubuntu-122.04, mixed mode, sharing)
Jenkins Version
2.428
Jenkins server is deployed on a linux machine as a service but behind nginx proxy.
Jenkins agent is deployed on a linux machine as a service.
Jenkins agent is throwing channel closed exception error, Jenkins agent is already connected with the Jenkins server but it suddenly closes the channel, because of which any job which is running on that server also fails.
Here are the logs of the agent when it disconnects and throws the channel closed exception,
INFO: Failed to synchronize IO streams on the channel hudson.remoting.Channel@52474407:big-server
hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@52474407:big-server": Remote call on big-server faile>
at hudson.remoting.Channel.call(Channel.java:996)
at hudson.remoting.Channel.syncIO(Channel.java:1735)
at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1408)
at jdk.internal.reflect.GeneratedMethodAccessor532.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:924)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:902)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@52474407:big-server": channel is already c>
at hudson.remoting.Engine$1AgentEndpoint.lambda$onClose$1(Engine.java:637)
... 6 more
Dec 14, 2023 5:32:10 AM hudson.remoting.Request$2 run
INFO: Failed to send back a reply to the request UserRequest:UserRPCRequest:hudson.Launcher$RemoteProcess.join[](55): hudson.r>
Dec 14, 2023 5:32:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
I tried increasing the proxy connect and proxy send timeout in nginx conf, if nginx is closing the web socket connection, but it wasn't an issue here.
I tried with weekly restarting of jenkins-agent service on the agent node for a new web socket connection, but it didn't resulted in an improvement or reduction in the above error.