I'm having a strange Redis issue that causes my redis db0 to get completely wiped periodically for no obvious reason.
I'm using redis with Bull queue for NestJS and inserting volatile keys with 7 days duration. Everything is hosted on Google Cloud Kubernetes.
As you can see from the logs, queue fills up steadily and all of a sudden its wiped to 0.
1:M 14 Dec 2022 18:02:53.786 - Accepted 10.132.15.228:46851
1:M 14 Dec 2022 18:02:55.818 * DB saved on disk
1:M 14 Dec 2022 18:02:55.836 - DB 0: 6539 keys (1179 volatil
...skipping 1 line
1:M 14 Dec 2022 18:02:55.836 . 25 clients connected (0 repli
cas), 10264320 bytes in use
1:M 14 Dec 2022 18:02:56.338 * DB saved on disk
1:M 14 Dec 2022 18:02:57.975 * DB saved on disk
1:M 14 Dec 2022 18:02:58.444 * DB saved on disk
1:M 14 Dec 2022 18:02:58.687 * DB saved on disk
1:M 14 Dec 2022 18:03:00.863 - DB 0: 10 keys (3 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:00.863 . 25 clients connected (0 repli
cas), 3440672 bytes in use
1:M 14 Dec 2022 18:03:01.056 * DB saved on disk
This part is out of time, like 2 threads are writing to same output:
1:M 14 Dec 2022 18:03:10.900 - DB 0: 12 keys (5 volatile) in ...back 1 page
1:M 14 Dec 2022 18:02:40.773 - DB 0: 6542 keys (1174 volatile) in 8192 slots HT.
Any insights would be greatly appreciated.
Server info:
redis_version:7.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:68a9a9d6a665b45c
redis_mode:standalone
os:Linux 5.10.133+ x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:ca394137bd91b23622a8facfb97ae86ba2325a01
tcp_port:6379
server_time_usec:1671042411041619
uptime_in_seconds:1752
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:10098026
executable:/data/redis-server
config_file:/redis-config/redis.conf
io_threads_active:0
# Clients
connected_clients:25
cluster_connections:0
Redis config:
save 60 1000
loglevel debug
notify-keyspace-events Ex
dbfilename dump.rdb
dir /data
logfile /data/redis.log
Debug log:
...skipping 1 line
1:M 14 Dec 2022 17:59:55.106 . 24 clients connected (0 repli
cas), 8023104 bytes in use
1:M 14 Dec 2022 18:00:00.125 - DB 0: 5552 keys (1125 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:00.126 . 24 clients connected (0 repli
cas), 7959688 bytes in use
1:M 14 Dec 2022 18:00:05.146 - DB 0: 5552 keys (1128 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:05.146 . 24 clients connected (0 repli
cas), 8038576 bytes in use
1:M 14 Dec 2022 18:00:10.166 - DB 0: 5553 keys (1128 volatil
...skipping 1 line
1:M 14 Dec 2022 18:00:10.166 . 24 clients connected (0 repli
cas), 7977384 bytes in use
1:M 14 Dec 2022 18:00:14.281 * 1000 changes in 60 seconds. S
aving...
1:M 14 Dec 2022 18:00:14.282 * Background saving started by
pid 26
26:C 14 Dec 2022 18:00:14.284 - Fork CoW for RDB: current 0
MB, peak 0 MB, average 0 MB
26:C 14 Dec 2022 18:00:14.378 * DB saved on disk
26:C 14 Dec 2022 18:00:14.378 * Fork CoW for RDB: current 0
MB, peak 0 MB, average 0 MB
...skipping 1 line
1:M 14 Dec 2022 18:00:15.187 - DB 0: 6081 keys (1128 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:15.187 . 24 clients connected (0 repli
cas), 9276960 bytes in use
1:M 14 Dec 2022 18:00:17.755 - Accepted 10.8.1.1:36758
1:M 14 Dec 2022 18:00:20.208 - DB 0: 6570 keys (1131 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:20.208 . 24 clients connected (0 repli
cas), 10252224 bytes in use
1:M 14 Dec 2022 18:00:25.227 - DB 0: 6570 keys (1133 volatil
e) in 8192 slots HT.
...skipping 1 line
1:M 14 Dec 2022 18:00:30.247 - DB 0: 6568 keys (1134 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:30.247 . 24 clients connected (0 repli
cas), 10266176 bytes in use
1:M 14 Dec 2022 18:00:35.267 - DB 0: 6568 keys (1137 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:35.267 . 24 clients connected (0 repli
cas), 10345096 bytes in use
1:M 14 Dec 2022 18:00:40.285 - DB 0: 6569 keys (1137 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:40.285 . 24 clients connected (0 repli
...skipping 1 line
1:M 14 Dec 2022 18:00:45.304 - DB 0: 6563 keys (1140 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:45.304 . 24 clients connected (0 repli
cas), 10315776 bytes in use
1:M 14 Dec 2022 18:00:50.325 - DB 0: 6563 keys (1140 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:50.325 . 24 clients connected (0 repli
cas), 10254504 bytes in use
1:M 14 Dec 2022 18:00:55.342 - DB 0: 6562 keys (1143 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:00:55.342 . 24 clients connected (0 repli
...skipping 1 line
1:M 14 Dec 2022 18:01:00.360 - DB 0: 6562 keys (1143 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:00.360 . 24 clients connected (0 repli
cas), 10209512 bytes in use
1:M 14 Dec 2022 18:01:05.381 - DB 0: 6562 keys (1146 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:05.381 . 24 clients connected (0 repli
cas), 10309016 bytes in use
1:M 14 Dec 2022 18:01:10.399 - DB 0: 6563 keys (1146 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:10.399 . 24 clients connected (0 repli
...skipping 1 line
1:M 14 Dec 2022 18:01:15.019 * 1000 changes in 60 seconds. S
aving...
1:M 14 Dec 2022 18:01:15.020 * Background saving started by
pid 27
27:C 14 Dec 2022 18:01:15.022 - Fork CoW for RDB: current 0
MB, peak 0 MB, average 0 MB
27:C 14 Dec 2022 18:01:15.085 * DB saved on disk
27:C 14 Dec 2022 18:01:15.086 * Fork CoW for RDB: current 0
MB, peak 0 MB, average 0 MB
1:M 14 Dec 2022 18:01:15.120 * Background saving terminated
with success
...skipping 1 line
1:M 14 Dec 2022 18:01:15.421 . 24 clients connected (0 repli
cas), 10212776 bytes in use
1:M 14 Dec 2022 18:01:20.441 - DB 0: 6553 keys (1149 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:20.441 . 24 clients connected (0 repli
cas), 10192304 bytes in use
1:M 14 Dec 2022 18:01:25.460 - DB 0: 6553 keys (1152 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:25.460 . 24 clients connected (0 repli
cas), 10189448 bytes in use
1:M 14 Dec 2022 18:01:30.479 - DB 0: 6553 keys (1152 volatil
...skipping 1 line
1:M 14 Dec 2022 18:01:30.479 . 24 clients connected (0 repli
cas), 10189448 bytes in use
1:M 14 Dec 2022 18:01:35.503 - DB 0: 6553 keys (1155 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:35.503 . 24 clients connected (0 repli
cas), 10186592 bytes in use
1:M 14 Dec 2022 18:01:40.523 - DB 0: 6553 keys (1155 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:40.523 . 24 clients connected (0 repli
cas), 10207064 bytes in use
1:M 14 Dec 2022 18:01:45.543 - DB 0: 6551 keys (1157 volatil
...skipping 1 line
1:M 14 Dec 2022 18:01:45.543 . 24 clients connected (0 repli
cas), 10202824 bytes in use
1:M 14 Dec 2022 18:01:50.561 - DB 0: 6551 keys (1157 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:50.561 . 24 clients connected (0 repli
cas), 10202824 bytes in use
1:M 14 Dec 2022 18:01:55.582 - DB 0: 6546 keys (1160 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:01:55.582 . 24 clients connected (0 repli
cas), 10173504 bytes in use
1:M 14 Dec 2022 18:02:00.603 - DB 0: 6546 keys (1161 volatil
...skipping 1 line
1:M 14 Dec 2022 18:02:00.603 . 24 clients connected (0 repli
cas), 10192880 bytes in use
1:M 14 Dec 2022 18:02:05.622 - DB 0: 6545 keys (1163 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:05.622 . 24 clients connected (0 repli
cas), 10169456 bytes in use
1:M 14 Dec 2022 18:02:10.645 - DB 0: 6546 keys (1167 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:10.645 . 24 clients connected (0 repli
cas), 10290056 bytes in use
1:M 14 Dec 2022 18:02:15.667 - DB 0: 6546 keys (1167 volatil
...skipping 1 line
1:M 14 Dec 2022 18:02:15.667 . 24 clients connected (0 repli
cas), 10208312 bytes in use
1:M 14 Dec 2022 18:02:20.690 - DB 0: 6546 keys (1167 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:20.690 . 24 clients connected (0 repli
cas), 10208312 bytes in use
1:M 14 Dec 2022 18:02:25.713 - DB 0: 6546 keys (1170 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:25.713 . 24 clients connected (0 repli
cas), 10184984 bytes in use
1:M 14 Dec 2022 18:02:30.731 - DB 0: 6545 keys (1171 volatil
...skipping 1 line
1:M 14 Dec 2022 18:02:30.731 . 24 clients connected (0 repli
cas), 10223352 bytes in use
1:M 14 Dec 2022 18:02:35.750 - DB 0: 6541 keys (1173 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:35.750 . 24 clients connected (0 repli
cas), 10154896 bytes in use
1:M 14 Dec 2022 18:02:40.773 - DB 0: 6542 keys (1174 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:40.773 . 24 clients connected (0 repli
cas), 10194824 bytes in use
1:M 14 Dec 2022 18:02:44.425 - Accepted 10.132.15.230:61177
...skipping 1 line
1:M 14 Dec 2022 18:02:45.793 - DB 0: 6541 keys (1176 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:45.793 . 24 clients connected (0 repli
cas), 10172512 bytes in use
1:M 14 Dec 2022 18:02:50.817 - DB 0: 6540 keys (1179 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:50.817 . 24 clients connected (0 repli
cas), 10250272 bytes in use
1:M 14 Dec 2022 18:02:53.786 - Accepted 10.132.15.228:46851
1:M 14 Dec 2022 18:02:55.818 * DB saved on disk
1:M 14 Dec 2022 18:02:55.836 - DB 0: 6539 keys (1179 volatil
...skipping 1 line
1:M 14 Dec 2022 18:02:55.836 . 25 clients connected (0 repli
cas), 10264320 bytes in use
1:M 14 Dec 2022 18:02:56.338 * DB saved on disk
1:M 14 Dec 2022 18:02:57.975 * DB saved on disk
1:M 14 Dec 2022 18:02:58.444 * DB saved on disk
1:M 14 Dec 2022 18:02:58.687 * DB saved on disk
1:M 14 Dec 2022 18:03:00.863 - DB 0: 10 keys (3 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:00.863 . 25 clients connected (0 repli
cas), 3440672 bytes in use
1:M 14 Dec 2022 18:03:01.056 * DB saved on disk
...skipping 1 line
1:M 14 Dec 2022 18:03:01.989 - Client closed connection id=2
9 addr=10.132.15.228:46851 laddr=10.8.1.41:6379 fd=32 name=
age=8 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=
0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=97 obl
=0 oll=0 omem=0 tot-mem=22272 events=r cmd=save user=default
redir=-1 resp=2
1:M 14 Dec 2022 18:03:05.882 - DB 0: 10 keys (3 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:05.882 . 24 clients connected (0 repli
cas), 3418912 bytes in use
1:M 14 Dec 2022 18:03:10.900 - DB 0: 12 keys (5 volatile) in
...back 1 page
1:M 14 Dec 2022 18:02:40.773 - DB 0: 6542 keys (1174 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:40.773 . 24 clients connected (0 repli
cas), 10194824 bytes in use
1:M 14 Dec 2022 18:02:44.425 - Accepted 10.132.15.230:61177
1:M 14 Dec 2022 18:02:44.639 - Reading from client: Connecti
on reset by peer
1:M 14 Dec 2022 18:02:45.793 - DB 0: 6541 keys (1176 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:45.793 . 24 clients connected (0 repli
cas), 10172512 bytes in use
...skipping 1 line
1:M 14 Dec 2022 18:02:50.817 . 24 clients connected (0 repli
cas), 10250272 bytes in use
1:M 14 Dec 2022 18:02:53.786 - Accepted 10.132.15.228:46851
1:M 14 Dec 2022 18:02:55.818 * DB saved on disk
1:M 14 Dec 2022 18:02:55.836 - DB 0: 6539 keys (1179 volatil
e) in 8192 slots HT.
1:M 14 Dec 2022 18:02:55.836 . 25 clients connected (0 repli
cas), 10264320 bytes in use
1:M 14 Dec 2022 18:02:56.338 * DB saved on disk
1:M 14 Dec 2022 18:02:57.975 * DB saved on disk
1:M 14 Dec 2022 18:02:58.444 * DB saved on disk
...skipping 1 line
1:M 14 Dec 2022 18:03:00.863 - DB 0: 10 keys (3 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:00.863 . 25 clients connected (0 repli
cas), 3440672 bytes in use
1:M 14 Dec 2022 18:03:01.056 * DB saved on disk
1:M 14 Dec 2022 18:03:01.756 * DB saved on disk
1:M 14 Dec 2022 18:03:01.989 - Client closed connection id=2
9 addr=10.132.15.228:46851 laddr=10.8.1.41:6379 fd=32 name=
age=8 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=
0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=97 obl
=0 oll=0 omem=0 tot-mem=22272 events=r cmd=save user=default
...skipping 1 line
1:M 14 Dec 2022 18:03:05.882 - DB 0: 10 keys (3 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:05.882 . 24 clients connected (0 repli
cas), 3418912 bytes in use
1:M 14 Dec 2022 18:03:10.900 - DB 0: 12 keys (5 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:10.900 . 24 clients connected (0 repli
cas), 3378320 bytes in use
1:M 14 Dec 2022 18:03:15.917 - DB 0: 12 keys (5 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:15.917 . 24 clients connected (0 repli
cas), 3378336 bytes in use
1:M 14 Dec 2022 18:03:20.936 - DB 0: 12 keys (5 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:20.936 . 24 clients connected (0 repli
cas), 3419296 bytes in use
1:M 14 Dec 2022 18:03:25.954 - DB 0: 12 keys (5 volatile) in
16 slots HT.
1:M 14 Dec 2022 18:03:25.954 . 24 clients connected (0 repli
cas), 3378304 bytes in use