Redis keyspace notifications subscriptions in distributed environment using ServiceStack

Question

We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).

This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace@0__:expired', that service will pick it and take action. That's fantastic...

... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.

I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?

Otherwise, is there a way to delay a message publishing?

Thanks!

Answer 1

As describe in https://redis.io/topics/notifications

very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.

So client should create separate connection to each node to get redis keyspace notification.

Answer 2

My understanding of your question: You need an event based unicast notification whenever a key is expired.

This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!

Solution: You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!

How does it work? Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.

Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!

Ref:

List pop operation: https://redis.io/commands/lpop

Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?