I am running a redis cluster with 6 nodes, 1 master/node and 1 slave/master. Dataset is around 60-80GB/node. We are experiencing a peculiar problem where every 16th minute of each hour, redis responds very slow. Usual response time is < 5ms, but at 16th minute, the avg response time shoots above 15s. Is there any background process in redis that runs hourly?
My redis.conf is as below:-
port 23000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 20000
cluster-slave-validity-factor 1
logfile redis.log
loglevel notice
slowlog-log-slower-than 1000
slowlog-max-len 64
latency-monitor-threshold 100
maxmemory-policy volatile-ttl
slave-read-only no
protected-mode no
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error no
rdbchecksum yes
dbfilename dump.rdb
dir /data/redis/redis-master
appendonly yes
appendfilename redis-staging-ao.aof