On demand horizontally scaling event driven architectures

Question

What is the best way to horizontally scale an event driven architecture when load increases?

1. Many people suggest using Kakfa as the message queue source for EDA however Kafka only allows one consumer in a consumer group per partition. Repartitioning especially during heavy load situations can be costly and time consuming.
2. Having many consumers in a consumer group that take work and acknowledge quickly would give some horizontal scaling but now message order needs to be considered as well as load completion.
3. With RabbitMQ queues can be created and deleted on the fly however that would require an additional orchestrator to help manage and distribute load.

Also none of this addresses the load balancing problem that comes with the territory.

Any help would be appreciated. Thanks

Answer 1

A bit late in answering but here goes,

Your reasoning that the scaling should occur at the message bus layer is not entirely correct. If we take an end to end scenario, increased load means increase in incoming request to the front end (the API layer). See reference event driven architecture in link below.

Assuming some form of auto-scaling present (Kubernetes replication factor, Amazon autoscaler) the front end will scale out to handle extra load. After initial pre-processing the service will post the event to the message queue in event driven architecture.

In Kafka specifically the topic partition is a unit of scale out since one producer can write to one partition. Typically you would define the number of partition in advance based on throughput of single partition.

As the reference article mentions if single partition throughput is p and you need t as throughput then you need t/p partitions.

If t is the throughput of normal expected load you can create provision in advance for 2x, 3x, 10x throughput then normal by creating as many partitions.

Typically throughput on a single partition is in excess of 10 Mb/s.