We are using Hazelcast as embedded in our spring boot application and our application is hosted on 12 tomcat servers forming a single cluster.
com.hazelcast.spi.properties.ClusterProperty.OPERATION_CALL_TIMEOUT_MILLIS value is set to 8000
What could be the usual reason for such delays that hazelcast instances within the cluster taking more than 8sec to communicate?
Exception=com.hazelcast.core.OperationTimeoutException GetOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-11-21 03:46:33.437. Start time: 2023-11-21 03:46:16.441. Total elapsed time: 16996 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-11-21 03:46:24.293. Invocation{op=com.hazelcast.map.impl.operation.GetOperation{serviceName='hz:impl:mapService'
firstInvocationTimeMs=1700534776441, firstInvocationTime='2023-11-21 03:46:16.441', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 01:00:00.000', target=
What will be implications of increasing the value of OPERATION_CALL_TIMEOUT_MILLIS say to 20000