org.infinispan.CLUSTER : ISPN000474: Error processing request 0@host-a-25374

Question

Infinispan finds the 2 nodes of the cluster - but exchanging data between the nodes is failing.

org.infinispan.CLUSTER : ISPN000474: Error processing request 0@host-a-25374

java.lang.NullPointerException: Cannot read the array length because "this.bytes" is nullat org.infinispan.marshall.core.AbstractBytesObjectInput.checkPosLength(AbstractBytesObjectInput.java:260)at org.infinispan.marshall.core.AbstractBytesObjectInput.readByte(AbstractBytesObjectInput.java:107)at org.infinispan.marshall.core.AbstractBytesObjectInput.readUnsignedByte(AbstractBytesObjectInput.java:113)at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:356)at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)at 
org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1524)at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1464)at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$1(JGroupsTransport.java:1682)at java.base/java.lang.Iterable.forEach(Iterable.java:75)at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1674)at org.jgroups.JChannel.up(JChannel.java:749)at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:939)at org.jgroups.protocols.FRAG2.up(FRAG2.java:161)at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)at org.jgroups.protocols.pbcast.GMS.up(GMS.java:859)at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:498)at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:722)at org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119)at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)at org.jgroups.protocols.Discovery.up(Discovery.java:314)at org.jgroups.protocols.RED.up(RED.java:119)at org.jgroups.protocols.TP.passBatchUp(TP.java:1210)at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:287)at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:276)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)at java.base/java.lang.Thread.run(Thread.java:833)

We are using

org.springframework.boot version 3.1.5
org.infinispan:infinispan-spring-boot3-starter-embedded:14.0.20.Final

Deployment on Tomcat 10 running within JAVA 17.

We are using UDP as transfer protocol (nearly the default jgroups config by infinispan):

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-5.2.xsd">
   
   <UDP bind_addr="${jgroups.bind.address,jgroups.udp.address:SITE_LOCAL}"
        bind_port="${jgroups.bind.port,jgroups.udp.port:0}"
        mcast_addr="${jgroups.udp.mcast_addr:201.0.0.1}"
        mcast_port="${jgroups.udp.mcast_port:46655}"
        tos="0"
        ucast_send_buf_size="1m"
        mcast_send_buf_size="1m"
        ucast_recv_buf_size="20m"
        mcast_recv_buf_size="25m"
        ip_ttl="${jgroups.ip_ttl:2}"
        thread_naming_pattern="pl"
        diag.enabled="${jgroups.diag.enabled:false}"
        bundler_type="transfer-queue"
        bundler.max_size="${jgroups.bundler.max_size:64000}"

        thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
        thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
        thread_pool.keep_alive_time="60000"

        thread_pool.thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}"
   />
   <RED/>
   <PING num_discovery_runs="3"/>
   <MERGE3 min_interval="10000"
           max_interval="30000"
   />
   <FD_SOCK2 offset="${jgroups.fd.port-offset:50000}"/>
   <FD_ALL3/>
   <VERIFY_SUSPECT2 timeout="1000"/>
   <pbcast.NAKACK2 xmit_interval="100"
                   xmit_table_num_rows="50"
                   xmit_table_msgs_per_row="1024"
                   xmit_table_max_compaction_time="30000"
                   resend_last_seqno="true"
   />
   <UNICAST3 xmit_interval="100"
             xmit_table_num_rows="50"
             xmit_table_msgs_per_row="1024"
             xmit_table_max_compaction_time="30000"
   />
   <pbcast.STABLE desired_avg_gossip="5000"
                  max_bytes="1M"
   />
   <pbcast.GMS print_local_addr="false"
               join_timeout="${jgroups.join_timeout:2000}"
   />
   <UFC max_credits="${jgroups.max_credits:4m}"
        min_threshold="0.40"
   />
   <MFC max_credits="${jgroups.max_credits:4m}"
        min_threshold="0.40"
   />
   <FRAG4 frag_size="${jgroups.frag_size:60000}"/>
</config>

Starting two servers by bootRun on local development everything works as excepted - what did we miss?

Within spring-boot 2.7.x we did not see any problems, after upgrading, we run into the error.

many thanks in advance!