I have a problem with one of the servers in a unicast cluster, for a few days a server had been disconnected on the Anypoint Platform, I explain the architecture a bit, we have 2 servers with Mule Runtime 4.3 both on AWS, the server the primary is running fine with no problem but the secondary server fails I don't know why i checked the log on the damaged server but I can't see anything to help me I put a small log below hope you can help me or give me some advice on how to get the right trace.
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Mule cluster setup properties >>>
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Cluster id: 12611154
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Cluster node id: 2
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Discovery SPI is not enabled.
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Reading backup data: false
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Multicast to discover cluster nodes disabled
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP to discover cluster nodes enabled
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP time to connect 20 seconds
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: TCP/IP discovery members:
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: - 0.0.0.19
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: - 0.0.0.12
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Using port 5701 for inbound connections
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Inbound request port autoincrement is enabled
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Using random port for outobund connections
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: No network interface restriction. Using all available interfaces
INFO 2022-12-07 18:38:22,707 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Message backup count: 1
INFO 2022-12-07 18:38:22,723 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Quorum enabled: false
INFO 2022-12-07 18:38:22,723 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Quorum size: 0
INFO 2022-12-07 18:38:22,723 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.config.ClusterConfigLogger: Mule cluster setup properties <<<
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/C:/MuleRuntime-Dev/mule/lib/opt/hazelcast-3.12.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
DEBUG 2022-12-07 18:43:31,509 [WrapperListener_start_runner] [processor: ; event: ] com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager: Node failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:138) ~[hazelcast-3.12.jar:3.12]
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:228) ~[hazelcast-3.12.jar:3.12]
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:207) ~[hazelcast-3.12.jar:3.12]
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:157) ~[hazelcast-3.12.jar:3.12]
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57) ~[hazelcast-3.12.jar:3.12]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.configureServerMode(HazelcastClusterManager.java:167) ~[mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.<init>(HazelcastClusterManager.java:114) ~[mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.createManager(HazelcastClusterManager.java:223) ~[mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.lambda$0(HazelcastClusterManager.java:263) ~[mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.create(HazelcastClusterManager.java:275) [mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterManager.create(HazelcastClusterManager.java:263) [mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterCoreExtension.createHazelcastManager(HazelcastClusterCoreExtension.java:152) [mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at com.mulesoft.mule.runtime.module.cluster.internal.HazelcastClusterCoreExtension.initialise(HazelcastClusterCoreExtension.java:129) [mule-module-cluster-ee-4.3.0-20210119.jar:4.3.0-20210119]
at org.mule.runtime.module.launcher.coreextension.DefaultMuleCoreExtensionManagerServer.initializeCoreExtensions(DefaultMuleCoreExtensionManagerServer.java:178) [mule-module-launcher-4.3.0-20210119.jar:4.3.0-20210119]
at org.mule.runtime.module.launcher.coreextension.DefaultMuleCoreExtensionManagerServer.initialise(DefaultMuleCoreExtensionManagerServer.java:91) [mule-module-launcher-4.3.0-20210119.jar:4.3.0-20210119]
at org.mule.runtime.module.launcher.MuleContainer.start(MuleContainer.java:249) [mule-module-launcher-4.3.0-20210119.jar:4.3.0-20210119]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.mule.runtime.module.reboot.internal.MuleContainerWrapper.start(MuleContainerWrapper.java:49) [mule-module-boot-ee-4.3.0-20210119.jar:4.3.0-20210119]
at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4514) [wrapper-3.5.45.jar:3.5.45]
ERROR 2022-12-07 18:43:31,525 [WrapperListener_start_runner] [processor: ; event: ] org.mule.runtime.module.launcher.MuleContainer:
********************************************************************************
Message : Invalid cluster configuration: Node failed to start!
Element : org.mule.runtime.module.launcher.coreextension.DefaultMuleCoreExtensionManagerServer@3c67d3c4 @ app
Element DSL : (None)
Error type : (None)
FlowStack : (None)
(set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
ERROR 2022-12-07 18:43:31,525 [WrapperListener_start_runner] [processor: ; event: ] org.mule.runtime.module.launcher.MuleContainer:
********************************************************************************
* A Fatal error has occurred while the server was running: *
* Invalid cluster configuration: Node failed to start! *
* (java.lang.RuntimeException) *
* *
* The error is fatal, the system will shutdown *
********************************************************************************
<-- Wrapper Stopped