I just setup a new cluster containing 10 nodes, 5 of which are controllers and 5 are brokers. I created a new topic on one of the brokers and am unable to consistently find it. It appears to hop around the different brokers in the cluster. I am unable to list or delete the topic and get an error that it cannot be found.
I wrong a python script to list topics on all brokers in the cluster and this way I am able to see it periodically.
I created the topic:
kafka-topics --bootstrap-server host1:9092 --topic replica-test --create --replication-factor=3 --partitions=3
I am unable to list it:
[user kafka]$ kafka-topics --bootstrap-server host1:9092 --list KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic
If I attempt to create it again I get a message indicating its already created:
[user kafka]$ kafka-topics --bootstrap-server host1:9092 --topic replica-test --create --replication-factor=3 --partitions=3 Error while executing topic command : Topic 'replica-test' already exists. [2024-02-05 14:53:28,710] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'replica-test' already exists.
Using a python script I can list topics on all brokers in the cluster. I can sometimes find it this way, but the elusive topic only remains on a single broker for so long, before it appears on another one.
First run:
`host1 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic
host2 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic
host3 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic replica-test `
Second run:
`host1 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic
host2 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic replica-test
host3 KAFKATEST __consumer_offsets chase-test mytopic testPubSub-StateTransfer testReqResp-StateRequest testtopic `
You can see it was on host3 but is now on host2. I am very confused as to what is happening here. Does anyone know why this is occurring and how I can create topics to avoid this?
I've tried deleting the topic and recreating the topic. Deleting the topic gives me a error and recreating the topic tells me it already exists.