We have application, that receives data from kafka, adds it to distributed redis (serverless) sorted set and a scheduler that runs every minute, reads data from sorted set and after processing publishes on kafka.
For scalability, if we add more instances with kafka partition key properly chosen, from kafka side we can manage with multiple instances. However we are not getting how we can have multiple instances dealing with same data structure operating in parallel.(something like kafka partitions)
Is there any way, we can have redis set corresponding to each instances, but if that instance goes away, that would remain unoperated that we dont want.