Akka design: How to add/remove routee from cluster aware router dynamically

Question

I have the following use case and I am not sure if the akka toolkit provide this out of the box:

• I have a number of nodes (instance/machine) that can run a finite number of long running task in the background and cannot accept more work while at max capacity.
• Each instance can only process 50 tasks.
• All instances are behind a load balancer.
• Each task can respond to messages from the client who initiated the task, since the client sends the messages via the load balancer the instances need to route it to the correct instance that handles the task.

I have tried initially cluster sharding, but there doesn't seem to be a way to cap the maximum number of shard regions/actors per node (= #tasks).

Then I tried it with a cluster aware router, which acts as a guard for accepting or rejecting work. This seems to work reasonable well, one problem is that once it reaches capacity I need to remove it as a routee and add it back once it has capacity again.

Is there something out of the box that supports this use case or should I carry on with the routing option and if so how can I achieve this?

I'll update the description if you have further questions or something is unclear.

Answer 1

Your scenario sounds like a good fit for the work pulling pattern. The gist of this pattern is:

• A master actor coordinates units of work among a number of worker actors.
• Workers register themselves to the master, meaning that workers can be added or removed dynamically.
• When the master receives work to be done, the master notifies the workers that work is available. Workers pull units of work when they're ready, do what needs to be done with their respective units of work, then ask the master for more work when they're finished.

To learn more about this pattern, read the following (the first two links are listed in the Akka documentation):

• The original post (by Derek Wyatt): http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2
• A follow-on post (by Michael Pollmeier): http://www.michaelpollmeier.com/akka-work-pulling-pattern
• An application of the pattern in a clustered environment with a cluster-aware router (by Ryan Tanner): https://www.conspire.com/blog/2013/10/akka-at-conspire-part-5-the-importance-of/