This problem always occurs after a few days that microservices communicate with kafka, I have 3 nodes, and for each microservice I use a group id on a specific topic. The error is as follows.
Unable connect to node with id 1:
Failed fetch messages from 1: NodeNotReadyError: Attempt to send a request to node which is not ready (node id 1).
Failed fetch messages from 2: [Error 7] RequestTimedOutError
Failed fetch messages from 1: [Error 7] RequestTimedOutError
Failed fetch messages from 2: [Error 7] RequestTimedOutError
Error sending JoinGroupRequest_v2 to node 1 [[Error 7] RequestTimedOutError] -- marking coordinator dead
Marking the coordinator dead (node 1)for group _message_alarm_ticket_app1.
Failed fetch messages from 3: [Error 7] RequestTimedOutError
Heartbeat failed: local member_id was not recognized; resetting and re-joining group
Heartbeat session expired - marking coordinator dead
Marking the coordinator dead (node 3)for group nce_alarms.
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
OffsetCommit failed for group nce_alarms due to group error ([Error 25] UnknownMemberIdError: nce_alarms), will rejoin
Auto offset commit failed: [Error 25] UnknownMemberIdError: nce_alarms
Describe Topic
root@dev-s-kafka1:/opt/kafka/bin# ./kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic nce_alarms
Topic: nce_alarms TopicId: zDniSSlUTgS4bWyXKPg5Zw PartitionCount: 8 ReplicationFactor: 3 Configs: segment.bytes=1073741824
Topic: nce_alarms Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,2,1
Topic: nce_alarms Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 3,2,1
Topic: nce_alarms Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: nce_alarms Partition: 3 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: nce_alarms Partition: 4 Leader: 1 Replicas: 1,3,2 Isr: 3,2,1
Topic: nce_alarms Partition: 5 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1
Topic: nce_alarms Partition: 6 Leader: 3 Replicas: 3,1,2 Isr: 3,2,1
Topic: nce_alarms Partition: 7 Leader: 1 Replicas: 1,2,3 Isr: 2,3,1
Environment
- aiokafka version 0.8.0
- kafka-python version 2.0.2:
- Kafka Broker version 3.0.0:
- Python version 3.9.16
If more information is needed please let me know, unfortunately I have only recently started working on this project, but I can ask my colleagues.
Thank you.