twemproxy (nutcracker) adding redis instance and keeping consistency

Question

I set up twemproxy (nutcracker) with 2 redis servers as backends including slaves, sentinel and failover.

As soon as I add another redis server some of the keys are not able to be read, probably due to twemproxy redirecting to another redis.

How do I add another redis instance without breaking the consistency?

I want to use the setup as a consistent and very fast database.

Here are my settings:

redis_cluster:
  auto_eject_hosts: false
  distribution: ketama
  hash: fnv1a_32
  listen: 127.0.0.1:6379
  preconnect: true
  redis: true
  servers:
    - 127.0.0.1:7004:1 redis_1
    - 127.0.0.1:7005:1 redis_2

I want to keep sharding a job of the server and be able to add instances. Do I need to use another setup?

Answer 1

Twemproxy can't do that. You can use Redis Cluster, or if you want to use Twemproxy you have to use a technique called presharding. Which is, start directly with, like, 32 or 64 instances or alike, even if them all run in the same host to start. Then start moving instances from one box to another in order to scale to multiple actual servers. The word to the right of the instances configured inside Twemproxy "redis_1" are used in order to hash, so that you can change IP address when you move instances, and still the hashing will be the same for that server.

Redis Cluster is release candidate 2 at this point. While it needs more testing and deployments to be battle tested as Redis is, it is already a viable product, so you may want to test it as well.