Is it possible to modify the number of Partitions and / or Replicas once a Kafka Partition has been created?

Question

I have been looking at the documentation for Conduktor, which is a GUI for Kafka.

https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count/

Having browsed around the GUI I can't find an obvious way to change the number of Partitions or Replicas (Replication Factor) of a Kafka Topic once it has been created.

It is easy to change these numbers when creating a topic, as it is part of the GUI window when creating a topic.

Here's a screenshot of that GUI.

Once the topic is created however, I cannot find any obvious way to change either the number of Partitions or Replication Factor.

Looking at the per-topic settings, the number of ISR can be changed. (ISR = In Sync Replica)

This suggests to me that there is no way to change the number of topics or partitions once a topic has been created.

Since I am not completely sure what is the situation here, I thought it best to ask the community. It might be that this is a complex operation which can only be performed using the CLI, and even then I suspect that it may be inadvisable to do so in production, because it may consume a lot of compute resources to do the re-partitioning, possibly leading to downtime.

As noted in the Conduktor documentation, it is better to get this right the first time rather than having to modify the cluster arrangement at a later time. That seems like sensible advice.

Answer 1

You are right, there are a few things to keep in mind when changing topic partitions at a later stage.

Ideally you choose the right number of partitions from the beginning. However, there are several reasons that make it necessary to adjust the partitions later on, for example, the system load may change over time.

As long as you are aware of the implications of an adjustment, there is nothing wrong with it.

I recently published a blog post on how to change partitions that shows how to do this using Kadeck (it is a free Kafka UI) with just a few clicks.

You can also change the replica configuration with a few clicks on the same topic configuration page in Kadeck.

I hope this helps you anyway!

Answer 2

kafka-reassign-partitions script can be used to increase replication factor and partitions, but it requires manual placement of each, so it's not as simple as a number input box on a GUI.

If someone created some type of drag/drop interface for replica placement / checkbox for replica & partition placement, then that would be neat.