I'm using Infinispan HOTROD into java application running on IBM Liberty app server with JDK8.
HOTROD client (lib 12.1.11.Final-redhat-00001 version) is implemented via `
org.infinispan.jcache.remote.JCachingProvider :
@PostConstruct
private void setUp() {
LOGGER.info("START [setUp] CACHE");
File conf = new File(System.getenv("CLIENT_HOTROD_FILE_PATH"));
URI uri = conf.toURI();
// Retrieve the system wide cache manager via org.infinispan.jcache.remote.JCachingProvider
javax.cache.CacheManager cacheManager = Caching.getCachingProvider("org.infinispan.jcache.remote.JCachingProvider")
.getCacheManager(uri, this.getClass().getClassLoader(), null);
this.cache = cacheManager.getCache(DATAGRIDKEY);
LOGGER.info("END [setUp] cache " + this.cache.getName() );
}
HOT-ROD client config file:
infinispan.client.hotrod.server_list=server1.x.xx.xxx:11222;server2.x.xx.xxx:11222;server3..x.xx.xxx:11222;server4..x.xx.xxx:11222
infinispan.client.hotrod.auth_username=user-app
infinispan.client.hotrod.auth_password=password
infinispan.client.hotrod.auth_realm=default
infinispan.client.hotrod.sasl_mechanism=SCRAM-SHA-512
infinispan.client.hotrod.client_intelligence=HASH_DISTRIBUTION_AWARE
HOT-ROD client config properties available
Otherwise the config server REDHAT Data Grid (8.2.3 version) is the followed:
{
"distributed-cache": {
"mode": "ASYNC",
"remote-timeout": 17500,
"state-transfer": {
"timeout": 60000
},
"encoding": {
"key": {
"media-type": "text/plain"
},
"value": {
"media-type": "application/x-protostream"
}
},
"locking": {
"concurrency-level": 1000,
"acquire-timeout": 15000,
"striping": false
},
"statistics": true
}
}
Sometimes when the application has some requests into the log application side, occurs the java.net.SocketTimeoutException: GetOperation issue as the followed stacktrace:
[2/15/22 17:24:39:445 CET] 00000573 HOTROD W org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation handleException ISPN004098: Closing connection [id: 0x9a1a8fb2, L:/10.0.20.160:55295 ! R:10.0.18.97/10.0.18.97:11222] due to transport error java.net.SocketTimeoutException: GetOperation{Cache-Name-Test, key=[B0x4A6C636F6D2E6475636174692E77612E..[110], flags=1, connection=10.0.18.97/10.0.18.97:11222} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:185)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:822)
The issue is more frequent when the app receives a massive GET KEY requests, but sometimes the issue is occurred during an unmassive load requests.
Could you know how to resolve the issue ? Do you know any suggestion ?
Thanks
I tried to edit the client config HOT-ROD to understand the behavior of the client, below I want to summarize the new configuration changes :
I noticed that changing
infinispan.client.hotrod.socket_timeout=15000millisecond. Whatching app log file I seen the same error, but currently, instead 60000 ms, is to 15000 ms the socketTimeoutException. With the new setting the thread request successfully completing the thread request even if the exception is logged.I think to understand that into the pool sometimes it seems there are connections died and the client search and try another connection alive into the pool retrying the GetOperation.
As written in
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperationclass code into library hot-rod:In a nutshell I need to keep alive connections into the pool, I don't know which is the property for Data Grid client HOT-ROD. The next step I would like to try to set
infinispan.client.hotrod.tcp_keep_alive=trueto test if the behavior is changed, after that I will update the post.