Per this kadek.com article,
Kafka partition limits
There are no hard limits on the number of partitions in Kafka clusters. But here are a few general rules:
- maximum 4000 partitions per broker (in total; distributed over many topics)
- maximum 200,000 partitions per Kafka cluster (in total; distributed over many topics)
- resulting in a maximum of 50 brokers per Kafka cluster
This reduces downtime in case something does go wrong. But beware: It should not be your goal to push these limits. In many “medium data” applications, you don’t need any of this.
the guideline is to have 50 brokers per cluster at max.
If an application cannot do horizontal sharding for some reason, does this mean that a single Kafka cluster based solution isn't suitable for cloud scale use? By cloud scale I mean pretty much an infinite number of producers and consumers.