I have a Grails App (Grails 2.4.5 | Groovy 2.3.11 | grails-cache 1.1.8 | grails-cache-ehcache 1.0.101) that is deployed in a dev environment on two servers (Tomcat 8.5.84 | JDK 1.8.0_341). I do not have access to the server configurations. I'm having some issues with cache replications when the application is deployed. One node (server) has some data, while the other node(server) has some other data. This causes data inconsistencies after each refresh, as the data returned can either come from node 1 or node 2. I have tried to turn off the Hibernate Cache Query, but it didn't resolve my issue. The application needs caching, so any help to figure out why the cache are not replicated will be extremely helpful.
Server log sample errors:
[Replication Thread] DEBUG distribution.RMICacheManagerPeerProvider - Lookup URL //ehcachePeerName.project.com:7055/sampleCacheName
2023-02-07 12:23:02,366 [Replication Thread] WARN distribution.RMIAsynchronousCacheReplicator - Unable to send message to remote peer. Message was: error marshalling arguments; nested exception is:
java.net.SocketException: Broken pipe (Write failed)
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Broken pipe (Write failed)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:159)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1915)
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:292)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:154)
... 5 more
Here are the different file configurations:
Datasource.groovy
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = false
cache.provider_class = 'net.sf.ehcache.hibernate.EhCacheProvider' //Outdated class
// cache.region.factory_class = 'net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory'
//Default from empty Grails 2.4.5 project
// cache.region.factory_class = 'org.hibernate.cache.SingletonEhCacheRegionFactory' // Hibernate 3
// cache.region.factory_class = 'org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory' // Hibernate 4
singleSession = true // configure OSIV singleSession mode
flush.mode = 'manual' // OSIV session flush mode outside of transactional context
// cache.region.factory_class = 'grails.plugin.cache.ehcache.hibernate.BeanEhcacheRegionFactory' // For Hibernate before 4.0
// cache.region.factory_class = 'grails.plugin.cache.ehcache.hibernate.BeanEhcacheRegionFactory4' // For Hibernate 4.0 and higher
}
ehcache.xml
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../conf/ehcache.xsd">
<!--
This is a copy of ehcache-failsafe.xml.
-->
<diskStore path="java.io.tmpdir"/>
<!--
Mandatory Default Cache configuration. These settings will be applied to caches
created programmtically using CacheManager.add(String cacheName)
-->
<defaultCache
eternal="false"
overflowToDisk="false"
diskPersistent="false"
timeToIdleSeconds="120"
timeToLiveSeconds="120"
maxElementsInMemory="100000"
maxElementsOnDisk="10000000"
diskExpiryThreadIntervalSeconds="120"
memoryStoreEvictionPolicy="LRU"
/>
</ehcache>
EhCacheConfig.groovy
//grails.cache.enabled = false
def baseurl = "//ehcachePeerName:ehcachePeerPort
String enableCacheReplicationProperty = ApplicationContext.config.get('com.project.cache.replication.enabled',
(Environment.current == Environment.DEVELOPMENT || Environment.current == Environment.TEST) ? 'false' : 'true')
Boolean enableCacheReplication = Boolean.valueOf(enableCacheReplicationProperty)
String rmiReplicatorCacheEventListenerFactory = 'rmiReplicatorCacheEventListenerFactory'
String chatCacheEventListenerFactory = 'chatCacheEventListenerFactory' //extends extends CacheEventListenerFactory and overrides CacheEventListener createCacheEventListener(Properties properties)
String chatHashesCacheEventListenerFactory = 'chatHashesCacheEventListenerFactory'
config = {
defaultCache {
eternal false
overflowToDisk false
diskPersistent false
maxElementsInMemory 100000
timeToLiveSeconds 120
timeToIdleSeconds 120
}
defaults {
eternal true
overflowToDisk false
diskPersistent false
maxElementsInMemory 100000
timeToLiveSeconds 0 //forever
timeToIdleSeconds 0 //forever
}
cache {
name 'staticAppDataCache'
eternal false
timeToLiveSeconds 60 * 60 * 24
timeToIdleSeconds 60 * 100
}
cacheEventListenerFactory {
name chatHashesCacheEventListenerFactory
}
cacheEventListenerFactory {
name chatCacheEventListenerFactory
className ChatCacheEventListenerFactory.class.canonicalName
}
cache {
name ChatCacheService.CHAT_HASHES_CACHE_NAME
}
cache {
name FilterCacheService.FILTER_CACHE_NAME
eternal false
timeToLiveSeconds 60 * 60 * 12
timeToIdleSeconds 60 * 60 * 12
}
cache {
name UserService.USER_ACTIVITY_CACHE_NAME
eternal false
timeToIdleSeconds UserService.USER_INACTIVE_CUTOFF_SECONDS
}
if (enableCacheReplication) {
cacheManagerPeerProviderFactory {
factoryType 'rmi'
peerDiscovery 'manual'
rmiUrl "${ehcachePeerPort}/${BrowserNotificationService.BROWSER_NOTIFICATION_CACHE_NAME}"
rmiUrl "${ehcachePeerPort}/${ChatCacheService.CHAT_CACHE_NAME}"
rmiUrl "${ehcachePeerPort}/${DashboardCacheService.DASHBOARDS_CACHE_NAME}"
rmiUrl "${ehcachePeerPort}/${FilterCacheService.FILTER_CACHE_NAME}"
rmiUrl "$ehcachePeerPort}/${UserService.USER_ACTIVITY_CACHE_NAME}"
timeToLive 'site'
}
cacheManagerPeerListenerFactory {
port Utilities.ehcacheListenerPort
}
cacheEventListenerFactory {
name rmiReplicatorCacheEventListenerFactory
factoryType 'rmi'
replicateAsynchronously true
}
cache {
name BrowserNotificationService.BROWSER_NOTIFICATION_CACHE_NAME
cacheEventListenerFactoryName rmiReplicatorCacheEventListenerFactory
}
cache {
name ChatCacheService.CHAT_CACHE_NAME
cacheEventListenerFactoryName rmiReplicatorCacheEventListenerFactory
cacheEventListenerFactoryName chatCacheEventListenerFactory
}
cache {
name DashboardCacheService.DASHBOARDS_CACHE_NAME
cacheEventListenerFactoryName rmiReplicatorCacheEventListenerFactory
}
cache {
name FilterCacheService.FILTER_CACHE_NAME
cacheEventListenerFactoryName rmiReplicatorCacheEventListenerFactory
}
cache {
name UserService.USER_ACTIVITY_CACHE_NAME
cacheEventListenerFactoryName rmiReplicatorCacheEventListenerFactory
}
} else {
cache {
name BrowserNotificationService.BROWSER_NOTIFICATION_CACHE_NAME
}
cache {
name ChatCacheService.CHAT_CACHE_NAME
cacheEventListenerFactoryName chatCacheEventListenerFactory
}
cache {
name DashboardCacheService.DASHBOARDS_CACHE_NAME
}
cache {
name FilterCacheService.FILTER_CACHE_NAME
}
cache {
name UserService.USER_ACTIVITY_CACHE_NAME
}
} // end environment block
} // grails.cache.config end
pom.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.project</groupId>
<artifactId>project</artifactId>
<version>35.1.6-SNAPSHOT</version>
<relativePath>../PROJECT/pom.xml</relativePath>
</parent>
<artifactId>project-web</artifactId>
<packaging>grails-app</packaging>
<name>project-web</name>
<properties>
<grails.version>2.4.5</grails.version>
<grails.oauth.version>2.6.1</grails.oauth.version>
<grails.spring-security.version>2.0-RC4</grails.spring-security.version>
<company.security.version>7.2.19</company.security.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-dependencies</artifactId>
<version>${grails.version}</version>
<type>pom</type>
<exclusions>
<exclusion>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-logging-log4j</artifactId>
</exclusion>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-test</artifactId>
<version>${grails.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-plugin-testing</artifactId>
<version>${grails.version}</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.spockframework</groupId>
<artifactId>spock-grails-support</artifactId>
<version>0.7-groovy-2.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-datastore-test-support</artifactId>
<version>1.0.2-grails-2.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>scaffolding</artifactId>
<version>2.1.2</version>
<scope>compile</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>cache</artifactId>
<version>1.1.8</version>
<scope>compile</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.company.grails.plugins.cache-ehcache</groupId>
<artifactId>cache-ehcache</artifactId>
<version>1.0.101</version>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.grails.plugins</groupId>
<artifactId>cache</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.grails.plugins</groupId>-->
<!-- <artifactId>hibernate4</artifactId>-->
<!-- <version>4.3.8.1</version>-->
<!-- <scope>runtime</scope>-->
<!-- <type>zip</type>-->
<!-- </dependency>-->
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>hibernate</artifactId>
<version>3.6.10.14</version>
<scope>runtime</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-datastore-simple</artifactId>
</exclusion>
<!-- <exclusion>-->
<!-- <groupId>org.grails</groupId>-->
<!-- <artifactId>grails-datastore-gorm</artifactId>-->
<!-- </exclusion>-->
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-datastore-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-async</artifactId>
</exclusion>
<exclusion>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
</exclusion>
<exclusion>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache</artifactId>
</exclusion>
<exclusion>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-bootstrap</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>database-migration</artifactId>
<version>1.4.0</version>
<scope>runtime</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>tomcat</artifactId>
<version>8.0.50</version>
<scope>provided</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>com.oracle.client</groupId>
<artifactId>ojdbc8</artifactId>
<version>19.6.0.0.200114</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.2.4</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>resources</artifactId>
<version>1.2.8</version>
<scope>runtime</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>quartz</artifactId>
<version>1.0.1</version>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>cache-headers</artifactId>
<version>1.1.7</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>cached-resources</artifactId>
<version>1.1</version>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.grails.plugins</groupId>
<artifactId>resources</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>export</artifactId>
<version>1.5</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>build-test-data</artifactId>
<version>2.2.2</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>fixtures</artifactId>
<version>1.3</version>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>spring-security-core</artifactId>
<version>${grails.spring-security.version}</version>
<scope>runtime</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-expression</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
</exclusion>
<exclusion>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails.plugins</groupId>
<artifactId>webxml</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- NAF Dependencies -->
<dependency>
<groupId>com.company.naf</groupId>
<artifactId>company-j2ee</artifactId>
<exclusions>
<exclusion>
<groupId>xalan</groupId>
<artifactId>xalan</artifactId>
</exclusion>
<exclusion>
<groupId>jaxen</groupId>
<artifactId>jaxen</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.xmlbeans</groupId>
<artifactId>xmlbeans</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- NAF Dependencies (END) -->
<!-- OAuth Dependencies -->
<dependency>
<groupId>com.company.security.oauth</groupId>
<artifactId>oauth-resource-server-plugin</artifactId>
<version>${company.security.version}</version>
<scope>runtime</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>com.company.naf</groupId>
<artifactId>company-common</artifactId>
</exclusion>
<!-- Only for applications using grails version greater than 2.4.2-->
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-dependencies</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.company.security.oauth</groupId>
<artifactId>grails-oauth-core</artifactId>
<version>${company.security.version}</version>
<exclusions>
<exclusion>
<groupId>org.grails</groupId>
<artifactId>grails-core</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.gmaven.runtime</groupId>
<artifactId>gmaven-runtime-1.8</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
</exclusion>
<exclusion>
<groupId>com.company.naf</groupId>
<artifactId>netjets-common</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
<exclusion>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- OAuth Dependencies (END)-->
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>spock</artifactId>
<version>0.7</version>
<scope>test</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>spock-grails-env-extensions</artifactId>
<version>0.2</version>
<scope>test</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>code-coverage</artifactId>
<version>1.2.7</version>
<scope>test</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- only if using maven; don't move these into BuildConfig.groovy -->
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>runtime-logging</artifactId>
<version>0.4</version>
<scope>compile</scope>
<type>zip</type>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.184</version>
<exclusions>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
<exclusions>
<exclusion>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.184</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-sqs-java-messaging-lib</artifactId>
<version>1.0.4</version>
<type>jar</type>
<exclusions>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- https://github.com/davidtinker/grails-cors
Grails plugin to add Cross-Origin Resource Sharing (CORS) headers for Grails applications.
These headers make it possible for Javascript code served from a different host to easily
make calls to the application.
-->
<dependency>
<groupId>org.grails.plugins</groupId>
<artifactId>cors</artifactId>
<version>1.3.0</version>
<scope>runtime</scope>
<type>zip</type>
<exclusions>
<exclusion>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-web</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</project>