Getting RedisConnectionFailureException intermittently

Question

Stacktrace

Error reading data from redis for key: key_name. 
Error: org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; 
nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out at 
org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:65) at
 org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42) at
  org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) at
   org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:192) at org.springframework.data.redis.connection.jedis.JedisConnection.doWithJedis(JedisConnection.java:827) at org.springframework.data.redis.connection.jedis.JedisConnection.doInvoke(JedisConnection.java:165) at org.springframework.data.redis.connection.jedis.JedisConnection.lambda$new$0(JedisConnection.java:74) at 
   org.springframework.data.redis.connection.jedis.JedisInvoker$Synchronizer.invoke(JedisInvoker.java:1018) at org.springframework.data.redis.connection.jedis.JedisInvoker.just
   (JedisInvoker.java:112) at org.springframework.data.redis.connection.jedis.JedisStringCommands.get(JedisStringCommands.java:57) at
    org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:279) at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis
    (DefaultValueOperations.java:58) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61) at
     org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191) at
     org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97) at org.springframework.data.redis.core.DefaultValueOperations.get
     (DefaultValueOperations.java:54) at com.company.service.CacheServiceImpl.getHashFromDsRedis(CacheServiceImpl.java:75) at
     com.company.service.CacheServiceImpl$$FastClassBySpringCGLIB$$b22ab586.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke
     (MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at
     org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.company.instrumentation.aspect.impl.GenericInstrumentationAspectImpl.doInstrumentation(GenericInstrumentationAspectImpl.java:55) 
     at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(
     DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at 
     org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at 
     org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at 
     org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at ...

RedisConfig.java

@Bean("jedisFactory")
public JedisConnectionFactory jedisFactory() {

    RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration();
    redisStandaloneConfiguration.setHostName(redisProperties.getHostName());
    redisStandaloneConfiguration.setPort(redisProperties.getPort());
    redisStandaloneConfiguration.setPassword(redisProperties.getPassword());

    JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
    jedisPoolConfig.setMinIdle(redisProperties.getMinIdle());
    jedisPoolConfig.setMaxIdle(redisProperties.getMaxIdle());
    jedisPoolConfig.setMaxTotal(redisProperties.getMaxTotal());
    jedisPoolConfig.setMaxWait(Duration.ofMillis(redisProperties.getMaxWaitTime()));

    JedisClientConfiguration jedisClientConfiguration = JedisClientConfiguration.builder()
            .connectTimeout(Duration.ofMillis(redisProperties.getConnectTimeout()))
            .readTimeout(Duration.ofMillis(redisProperties.getReadTimeout())).usePooling().poolConfig(jedisPoolConfig).build();

    return new JedisConnectionFactory(redisStandaloneConfiguration, jedisClientConfiguration);
}

@Bean("redisTemplate")
public RedisTemplate<String, Object> redisTemplate(
        @Qualifier("jedisFactory") JedisConnectionFactory jedisConnectionFactory) {
    RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
    redisTemplate.setConnectionFactory(jedisConnectionFactory);
    redisTemplate.setKeySerializer(new StringRedisSerializer());
    redisTemplate.setValueSerializer(new StringRedisSerializer());
    redisTemplate.setHashKeySerializer(new StringRedisSerializer());
    redisTemplate.setHashValueSerializer(new StringRedisSerializer());
    redisTemplate.afterPropertiesSet();
    return redisTemplate;
}

Config values:

REDIS_CONNECT_TIMEOUT_MS="1000"
REDIS_POOL_MAX_IDLE="40"
REDIS_POOL_MAX_TOTAL="55"
REDIS_POOL_MAX_WAIT_TIME_MS="15"
REDIS_POOL_MIN_IDLE="15"
REDIS_READ_TIMEOUT_MS="200"

We are seeing RedisConnectionFailureException intermittently as mentioned in above stacktrace, quantum is around 2-3 per 15 mins, which is very low, but still it should not come ideally. As you can see we have set the connection-timeout = 1000ms & read-timeout=200ms, which is high compare to the response time we have from redis calls.

Fix tried

We reduced REDIS_POOL_MIN_IDLE a little as we thought that it might happen that older ideal connections are getting dropped by redis server after few seconds, & when request comes spring assigns these thread which has a dropped connection attached to it, which could result in RedisConnectionFailureException. Errors count have reduced somewhat, but not completely.

How can I debug this further or tune any app or server side config to see why's this happening ?