RedisConnectionFailureException intermittently

Question

We are observing RedisConnectionFailureException intermittently as mentioned in above stacktrace, quantum is around 3/15 mins, which is very low, but they should not come as latency from redis server end is very less (around ~ 5-20 ms).

Stacktrace:

Error reading data from redis for key: key_name. 
Error: org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; 
nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out at 
org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:65) at
 org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42) at
  org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) at
   org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:192) at org.springframework.data.redis.connection.jedis.JedisConnection.doWithJedis(JedisConnection.java:827) at org.springframework.data.redis.connection.jedis.JedisConnection.doInvoke(JedisConnection.java:165) at org.springframework.data.redis.connection.jedis.JedisConnection.lambda$new$0(JedisConnection.java:74) at 
   org.springframework.data.redis.connection.jedis.JedisInvoker$Synchronizer.invoke(JedisInvoker.java:1018) at org.springframework.data.redis.connection.jedis.JedisInvoker.just
   (JedisInvoker.java:112) at org.springframework.data.redis.connection.jedis.JedisStringCommands.get(JedisStringCommands.java:57) at
    org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:279) at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis
    (DefaultValueOperations.java:58) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61) at
     org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191) at
     org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97) at org.springframework.data.redis.core.DefaultValueOperations.get
     (DefaultValueOperations.java:54) at com.company.service.CacheServiceImpl.getHashFromDsRedis(CacheServiceImpl.java:75) at
     com.company.service.CacheServiceImpl$$FastClassBySpringCGLIB$$b22ab586.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke
     (MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.company.instrumentation.aspect.impl.GenericInstrumentationAspectImpl.doInstrumentation(GenericInstrumentationAspectImpl.java:55) 
     at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
     DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at 
     org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at ...

Our RedisConfig.java

@Bean("jedisFactory")
public JedisConnectionFactory jedisFactory() {

    RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration();
    redisStandaloneConfiguration.setHostName(redisProperties.getHostName());
    redisStandaloneConfiguration.setPort(redisProperties.getPort());
    redisStandaloneConfiguration.setPassword(redisProperties.getPassword());

    JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
    jedisPoolConfig.setMinIdle(redisProperties.getMinIdle());
    jedisPoolConfig.setMaxIdle(redisProperties.getMaxIdle());
    jedisPoolConfig.setMaxTotal(redisProperties.getMaxTotal());
    jedisPoolConfig.setMaxWait(Duration.ofMillis(redisProperties.getMaxWaitTime()));

    JedisClientConfiguration jedisClientConfiguration = JedisClientConfiguration.builder()
            .connectTimeout(Duration.ofMillis(redisProperties.getConnectTimeout()))
            .readTimeout(Duration.ofMillis(redisProperties.getReadTimeout())).usePooling().poolConfig(jedisPoolConfig).build();

    return new JedisConnectionFactory(redisStandaloneConfiguration, jedisClientConfiguration);
}

@Bean("redisTemplate")
public RedisTemplate<String, Object> redisTemplate(
        @Qualifier("jedisFactory") JedisConnectionFactory jedisConnectionFactory) {
    RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
    redisTemplate.setConnectionFactory(jedisConnectionFactory);
    redisTemplate.setKeySerializer(new StringRedisSerializer());
    redisTemplate.setValueSerializer(new StringRedisSerializer());
    redisTemplate.setHashKeySerializer(new StringRedisSerializer());
    redisTemplate.setHashValueSerializer(new StringRedisSerializer());
    redisTemplate.afterPropertiesSet();
    return redisTemplate;
}

Config values:

REDIS_CONNECT_TIMEOUT_MS="1000"
REDIS_POOL_MAX_IDLE="40"
REDIS_POOL_MAX_TOTAL="55"
REDIS_POOL_MAX_WAIT_TIME_MS="15"
REDIS_POOL_MIN_IDLE="15"
REDIS_READ_TIMEOUT_MS="200"

As you can see we have set the connection-timeout = 1000ms & read-timeout=200ms, which is high compare to the response time we have from redis calls (~ 10-20 ms)

Solution tried

We reduced REDIS_POOL_MIN_IDLE a little as we thought that it might happen that older ideal connections are getting dropped by redis server after few seconds, & when request comes spring assigns these thread which has a dropped connection attached to it, which could result in RedisConnectionFailureException. Errors count have reduced somewhat, but not completely.

Suspection

In DEBUG logs we could see that whenever there's a redis call made from application. These messages are getting logged.

DEBUG org.springframework.data.redis.core.RedisConnectionUtils: Fetching Redis Connection from RedisConnectionFactory
DEBUG org.springframework.data.redis.core.RedisConnectionUtils: Closing Redis Connection.

Is there a way with which we can monitor/plot on grafana and see what's happening exactly with the connections in the JedisPool, are they even working or making connection and closing on every hit.