Reading all offsets kafka in python

Question

I am trying to read all messages in the topic of Kafka. I am using confluent cloud service so don't run Kafka in my localhost. I set the configurations as: 'enable.auto.commit': 'True','auto.offset.reset': 'earliest', 'default.topic.config': {'auto.offset.reset': 'smallest'}. However it gives me no message, or if I send message from producer at the same time it gives only that message not all offset messages.

How can I read all offset messages in python?

Answer 1

I didn't use confluent cloud service, If you want to consume all offsets, there are a few things you should pay attention to.

1. set new consumer groupId that does not consume any data
2. set 'auto.offset.rese=earliest' or 'auto.offset.reset=smallest' , you need to refer to the kafka version
3. pay attention to the automatic expiration date of the Topic