I am using kafka for job processing, Jobs usually take from somewhere around 2 min to 30,40 minutes. For that I have increased the kafka poll time out and also increased some other timeouts in the configuration. But with this below configuration I am getting this error exactly after 10 minutes of message processing
KafkaConnectionError: Socket EVENT_READ without in-flight-requests
WARNING, "func": "_conn_state_change", "lineno": "331", "msg": "Node 2 connection failed -- refreshing metadata"
WARNING, "func": "_run_once", "lineno": "992", "msg": "Heartbeat poll expired, leaving group"}
INFO "func": "maybe_leave_group", "lineno": "773", "msg": "Leaving consumer group
Here is my kafka consumer config
kafka_consumer = KafkaConsumer(
bootstrap_servers=[env.kafka_broker_address],
auto_offset_reset="earliest",
group_id=env.kafka_consumer_group,
value_deserializer=lambda x: json.loads(x.decode("utf-8")),
enable_auto_commit=False,
max_poll_interval_ms=2500000,
max_poll_records=2,
heartbeat_interval_ms=1000,
request_timeout_ms=3000000,
session_timeout_ms=1000000,
connections_max_idle_ms=3300000,
)
Is there something I am missing in kafka consumer configs?