We are running Infinispan 7.2.5 with 3 instances in REPL cluster. Spark Client is connected to the cluster using HotRod.
Suddenly, the view is updated and one instance is removed from the cluster and from client view as well. The instance was up & running but was not able to connect to other instances and was giving timed out exception.
What could have caused the instance to leave the cluster and not let it join back ? If someone can provide some insights on this.
Meanwhile, CPU was too high on the instance. Is it because of re-join attempts(if it actually takes much of CPU) or something else could be the reason ?
Client connected to Infinispan using HotRod observed below excpetion :
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
During the sametime, Infinispan server instance received below logs :
WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p3-t13) ISPN000071: Caught exception when handling command StateResponseCommand{cache=AsrlEnbTopologyCache, origin=asr-1-asrltopologyservice-24247, topologyId=9}: org.infinispan.util.concurrent.TimeoutException: Timed out applying state
at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:542) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:62) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
whereas, other instance available which seems to be separated from the cluster has received below logs:
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t12) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest@6ac2f8a3, req_id=1067, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t14) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest@1984efac, req_id=1069, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more