We have 3 Nodes of Ignite 2.7.6 cluster where we defined off heap memory/data region "datawarm" with max size 10GB and persistence enabled. We are facing very strange issue for more than a month. While connecting via SQLLine or Java Thin Client API, suddenly out of three nodes only 2 or sometime 1 node only providing the response (random order each time). After restart of the cluster the issue got resolved each time however after 3-4 hours of restart again it started. During checking the logs on the node where the connection not established only found below. We don't have any clue how to resolve this.
Jul 21 18:08:13 node1.example.com Ignite[6097]: 2020-07-21 18:08:13,219 DEBUG c.e.d.Logger [grid-nio-worker-client-listener-0-#30] Got client connection from address: /10.0.0.12:42262