Any workaround for distributing(sharding) key of collections like Lists, Sets etc

Question

We are using Redis 2.8.17 as JobQueues.

We are using RPUSH and BRPOPLPUSH for making a reliable queue.

As per our current design multiple app-servers push(RPUSH) jobs to a single job queue. Since the BRPOPLPUSH operation is atomic in redis, the jobs will later be poped(BRPOPLPUSH) and processed by any of the server's consumers.

Since the app servers are capable of scaling out, I am bit concerned that REDIS might become bottleneck in the future.

I learnt the following from documentation on redis partitioning: "it is not possible to shard a dataset with a single huge key like a very big sorted set"

I wonder whether pre-sharding the queues for app servers is the only option to scale out.

Is there anything that cluster can do in the above design?

Answer 1

The main thing you need to consider is whether you'll need to shard at all. The entire StackExchange network (not just StackOverflow -- all the network) runs off of 2 Redis servers (one of which I'm pretty sure only exists for redundancy), which it uses very aggressively. Take a look at http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/

Redis is absurdly fast (and remarkably space-efficient), with only one caveat: deleting an entire list/set/sorted set/hash is O(n), where n is the number of elements it contains. As long as you don't do that (operations like RPOP, BRPOP, BRPOPLPUSH, etc. don't count -- those are constant time), you should be golden.

TLDR: Unless you're planning to go bigger than StackOverflow, you won't need to shard, especially for a simple job queue.