ehcache RMI replication: java.rmi.NoSuchObjectException: no such object in table

Question

I've been looking for a solution online but could not find one to solve the issue I have with an ehcache replication. I'm using the manual rmi replication between 2 servers (server 1 = A.A.A.A and server 2 = B.B.B.B) and I'm getting this error:

2018-09-04 00:07:40,517 DEBUG [ehcache.distribution.RMICacheManagerPeerProvider] (req:) (Replication Thread:null) Lookup URL //B.B.B.B:40000/AddressDao
2018-09-04 00:07:40,522 WARN  [ehcache.distribution.RMIAsynchronousCacheReplicator] (req:) (Replication Thread:null) Unable to send message to remote peer.  Message was: no such object in table
java.rmi.NoSuchObjectException: no such object in table
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
    at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)

Versions

• ehcache: 2.10.5
• spring: 4.3.9.RELEASE
• JDK: 8

Configurations

Ehcache

The ehcache.xml file looks as follow:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="ehcache.xsd">
<cacheManagerEventListenerFactory
    class=""
    properties=""/>
<cacheManagerPeerProviderFactory
 class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
    properties="peerDiscovery=manual"
    propertySeparator="," />
<cacheManagerPeerListenerFactory
   class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
    properties="port=40000,remoteObjectPort=40001,hostName=A.A.A.A" />
<defaultCache
    maxElementsInMemory="10000"
    eternal="false"
    timeToIdleSeconds="120"
    timeToLiveSeconds="120"
    overflowToDisk="false"
    diskSpoolBufferSizeMB="30"
    maxElementsOnDisk="10000000"
    diskPersistent="false"
    diskExpiryThreadIntervalSeconds="120"
    memoryStoreEvictionPolicy="LRU"
    />
</ehcache>

Spring beans

The RMI server is configured with Spring for ehcache 2.5+:

<bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
    <property name="configLocation" value="classpath:ehcache.xml"/>
    <property name="shared" value="false"/>
    <property name="acceptExisting" value="true"/>
</bean>
<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean">
    <property name="locateExistingServerIfPossible" value="true"/>
</bean>

Java code

Cache creation

To create the caches, here is the code snippet:

    final CacheManager cm = CacheManager.create();
    _cache = cm.getCache(getName()); // If the cache is configured in the ehcache XML

    if (_cache == null) {
        final int maxElements = NumbersUtil.parseInt(value, 0);
        final int live = NumbersUtil.parseInt((String)params.get("cache.time.to.live"), 300);
        int idle = NumbersUtil.parseInt((String)params.get("cache.time.to.idle"), 300);
        if (live > 0 && idle > live) {
            idle = live;
        }
        _cache = new Cache(getName(), maxElements, false, live == 0, live, idle);

        // Check if replication is enabled and add the replication if needed
        final CacheManagerPeerListener cacheManagerPeerListener = cm.getCachePeerListener("RMI");
        final CacheManagerPeerProvider cmPeerProvider = cm.getCacheManagerPeerProvider("RMI");
        if (cacheManagerPeerListener != null && cmPeerProvider != null) {
            _cache.getCacheEventNotificationService().registerListener(new RMIAsynchronousCacheReplicator(false, false, true, false, true, 1000, 1000));
        }
        cm.addCache(_cache);
        s_logger.info("Cache created: " + _cache.toString());
    }

Cache replication registration

Code when a server is joining the cluster:

public void onServerJoined(List<? extends Server> nodeList, long selfNodeId) {
    final CacheManager cm = CacheManager.create();
    CacheManagerPeerProvider peerProvider = cm.getCacheManagerPeerProvider("RMI");
    if (peerProvider != null) {
        for (Server host : nodeList) {
            if (selfNodeId == host.getId()) {
                continue;
            }
            for (String cacheName : cm.getCacheNames()) {
                final String peerUrl = "//" + host.getServiceIP() + ":40000/" + cacheName;
                peerProvider.registerPeer(peerUrl);
            }
        }
    }
}

 Troubleshooting

Using this Java code, the cache can be listed:

$ java RmiPortNamesDisplay A.A.A.A 40000
Names bound to RMI registry at host A.A.A.A and port 40000:
    AddressDao
$ java RmiPortNamesDisplay B.B.B.B 40000
Names bound to RMI registry at host B.B.B.B and port 40000:
    AddressDao

Ports 40000 and 400001 are opened on the firewall of each server.

Playing with the ehcache JMX manager, if I remove all entries from the cache, the exception (at the top) is thrown and the cache isn't cleared on the other server.

Does someone has encountered this issue or has a solution, a hint?

Answer 1

I finally found what was causing this issue. The JVM was started with -Djava.rmi.server.hostname=127.0.0.1 which did not let the RMI server bind correctly. You can either remove this argument or set it to the IP used by ehcache to do the replication.