Bug / Change in transactional behaviour between Infinispan 9.4.20.Final to 10.1.8.Final

Question

We are using Infinispan to control a distributed cache (replicated-cache) in an JEE application running on a Payara server (Enterprise v 5.22.0) with Java 8 (OpenJDK 64-Bit Server VM Vendor: Azul Systems, Inc. Version: 25.262-b19)

In order to have a controlled start up of the application when starting multiple instances we have created a PESSIMISTIC locking cache, called Mutex, that is used to "lock" the cluster to allow 1 instance to load the cache while the others wait. The first instance to the get the lock reads a database and loads many other caches which are all configured as OPTIMISTIC locking. These cache "puts" all happen inside the outer Mutex transaction. The OPTMISTIC caches are defined with state-transfer enabled=true so that when the instance loading the cache from the database is done and releases the Mutex lock by committing the outer transaction the caches are updated on all instances too.

When loading the OPTIMISTIC caches we sometimes use entries in CACHE1 to drive the loading of CACHE2 (we do have more meaningful names but that detail does not matter here). So having loaded CACHE1 we use CACHE1.values() to orchestrate entries into CACHE2.put().

Now to the problem...

At V9.4.20.Final (and below) the process above works. At V10.x (also V.11.0.5.Final) this does not work. We have debugged our code to find that at V10.x the entries written to CACHE1 (all caches are isolation="READ_COMMITTED") are not visible with CACHE1.values() when trying to load CACHE2. Just to confirm this same code works at V9 where CACHE1.values() does return the values as expected as it is in the same transaction and should be able to see the entries.

If at V10 we don't have the outer Mutex transaction or commit the outer Mutex transaction before trying to read CACHE1 then all works.

The question:

Has the transactionality changed to remove visibilty of entries written in a nested transaction to the process that wrote them?

I have tried Weblogic 12.2.13 suspecting that the containers transaction manager might behave differently, but no. It fails at V10 works with V9 on Weblogic.

I can provided full code reproducer in a zip (eclipse / gradle project) but here are code snippets:

The CacheServiceImpl has a method exclusivePutAndGetAll and locks with name LOCK_KEY which can be called with a boolean to control whether entries are read before or after the "Mutex" parent transaction is committed:

@Override
    public <K, V> Collection<V> exclusivePutAndGetAll(String cacheName, Map<K, V> values, boolean insideMutex) throws Exception {

        Collection<V> returnValues = null;

        LOGGER.debug("mutex manager is " + mutexManager.getManagerHash());
        LOGGER.debug("cache manager is " + cacheManager.getManagerHash());

        LOGGER.info("Acquiring mutex lock before starting context");
        mutexManager.startTransactionAndAcquireMutex("LOCK_KEY");

        putAll(cacheName, values);

        if (insideMutex) {
            returnValues = getAll(cacheName); // this only works and returns values with V9 !!
        }

        mutexManager.commitTransaction();
        LOGGER.info("Mutex lock released after starting context.");

        if (!insideMutex) {
            returnValues = getAll(cacheName); 
        }

        return returnValues;
    }

And here's the mutexManager's startTransactionAndAcquireMutex which begins the transaction and locks the cache called Mutex with the provided "LOCK_KEY"

@Override
    public boolean startTransactionAndAcquireMutex(String mutexName) {
        final TransactionManager transactionManager = mutexCache.getTransactionManager();
        LOGGER.debug("Mutex cache TransactionManager is " + transactionManager.getClass());

        try {
            transactionManager.begin();
        } catch (NotSupportedException | SystemException ex) {
            throw new CacheException("Unable to start transaction for mutex " + mutexName, ex);
        }
        return acquireMutex(mutexName);
    }

and here is the mutexManager aquiring the lock:

@Override
    public boolean acquireMutex(String mutexName) {
        final TransactionManager transactionManager = mutexCache.getTransactionManager();

        boolean lockResult = false;
        try {
            if (transactionManager.getStatus() == Status.STATUS_ACTIVE) {
                lockResult = mutexCache.lock(mutexName);
            }
        } catch (final SystemException ex) {
            throw new CacheException("Unable to lock mutex " + mutexName, ex);
        }

        return lockResult;
    }

and finally the cache configuration in XML

<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="urn:infinispan:config:10.1 http://www.infinispan.org/schemas/infinispan-config-10.1.xsd" 
    xmlns="urn:infinispan:config:10.1">

    <jgroups>
        <stack-file name="tcp-cdl" path="cluster-jgroups.xml"/>
    </jgroups>

    <cache-container name="SampleCacheManager" statistics="true">
        <transport stack="tcp-cdl"/>
        <jmx duplicate-domains="true"/>
        
        <replicated-cache-configuration name="replicated-cache-template" statistics="true" mode="SYNC" remote-timeout="120000">
            <locking isolation="READ_COMMITTED" acquire-timeout="120000" write-skew="false" concurrency-level="150" striping="false"/>
            <state-transfer enabled="true" timeout="240000" chunk-size="10000"/>
            <transaction
                transaction-manager-lookup="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
                mode="NON_XA"
                locking="OPTIMISTIC">
            </transaction>   
        </replicated-cache-configuration>
        
        <replicated-cache name="Mutex" configuration="replicated-cache-template">
            <transaction locking="PESSIMISTIC" />
        </replicated-cache>   
        
    </cache-container>
</infinispan>