I will explain a little about the architecture that we have the problem, we have a cluster in AWS with 2 servers, one is the primary and the other is the replica in on premise version 4.3 (both servers running in AWS) with JVM 11.0.12, but I don't know why for days the secondary instance failed and the Mule runtime has appeared as in "disconnected" state, I decided to enter the server to check and I found in the mule logs the following message: A Fatal error has occurred while the server was running:
*
* Invalid cluster configuration: Node failed to start! *
* (java.lang.RuntimeException)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:hazelcast-3.12.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
DEBUG 2022-12-07 18:43:31,509 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager: Node failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:138) ~[hazelcast-3.12.jar:3.12]
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Mule cluster setup properties >>>
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Cluster id: 12611154
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Cluster node id: 2
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Discovery SPI is not enabled.
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Reading backup data: false
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Multicast to discover cluster nodes disabled
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP to discover cluster nodes enabled
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP time to connect 20 seconds
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP discovery members:
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: - 10.0.3.193
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: - 10.0.3.84
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Using port 5701 for inbound connections
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Inbound request port autoincrement is enabled
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Using random port for outobund connections
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: No network interface restriction. Using all available interfaces
INFO 2022-12-07 16:07:44,694 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Message backup count: 1
INFO 2022-12-07 16:07:44,710 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Quorum enabled: false
INFO 2022-12-07 16:07:44,710 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Quorum size: 0
INFO 2022-12-07 16:07:44,710 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Mule cluster setup properties <<<
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/C:/MuleRuntime-Dev/mule/lib/opt/hazelcast-3.12.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
DEBUG 2022-12-07 16:12:49,744 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager: Node failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:138) ~[hazelcast-3.12.jar:3.12]
and when we try to reset the server or the instance manually the server show the same message I don't know what happen because nobody changed anything.
Could you please help me if you had been the same error or know how to solve.
This is probably caused by a connectivity issue between the nodes or some change in the network that prevents a node to reach the other or changed the access (like change of IPs, ports closed, etc). Another possibility is a change in Hazelcast libraries in only one of the nodes but you said you didn't make changes and you should not change those libraries unless provided in a Mule patch.
Review the logs for other clustering errors. Don't pay attention to debug level logs. Try to identify what the problem was and fix the networking in your AWS VPC.
The warning
An illegal reflective access operation has occurredshould not relevant to this error. It is fixed in Mule 4.4.0. It happens when running in previous Mule 4.x releases with JDK 11.