Redis how to listen changes in 1000 to 10,000 lists using golang?

Question

I have 1000 to 10,000 keys stored in Redis, their value type is list. When a new item is added to any one of the existing lists, I need my golang program to be notified. Once notification is received I need to spawn a new goroutine and perform a small operation.

I am using redigo for redis connection pool.

What is the best approach to solve this problem, without overloading the Redis instance?

Answer 1

You can enable Redis' keyspace notifications and subscribe to the relevant events on the keys/patterns that interest you.

More details can be found in the documentation: http://redis.io/topics/notifications

Answer 2

I haven't tried this, but as speculation, I would use Redis's Lua to implement 2 new commands - MSR_PUSH and MSR_POP - which would do the following respectively:

-- MSR_PUSH
redis.call("PUSH", KEYS[1], ARGV[1])
redis.call("PUBLISH", "notify", KEYS[1])

and:

-- MSR_POP
local v = redis.call("POP", KEYS[1])
if v then
  redis.call("PUBLISH", "notify", KEYS[1])
end
return v

So, these Lua scripts update the lists as you normally do, but then also publish the keyname that was updated to the notify pub/sub, which will then allow a watching script (golang) to do something. You could also just push to another queue, and long poll that.

This link has more information on Lua with Redis: https://www.redisgreen.net/blog/intro-to-lua-for-redis-programmers/