The data node "Unreachable" cannot be restored to "healthy" after

Question

The data node "Unreachable" cannot be restored to "healthy" after version:cnosdb 2.4.0, revision 2fd7e8d02516beb696472d9bee115d948c224d1c

start 3m2d cluster.

show datanode node,check the status;

kill one of the datanode ;

wait a litter minutes;

show datanode node,check the status; one of the datanode was 'Unreachable';

start the data node which was killed;

wait a litter minutes;

show datanode node,check the status; one of the datanode was 'Unreachable';

cannot recover healthy; and check the meta log:

2024-02-01T10:11:58.526608000Z  INFO meta::store::storage: METADATA WRITE(ver: 1046): /cluster_xxx/data_nodes_metrics/2001 :{"id":2001,"disk_free":23970435072,"disk_total":243107192832,"time":1706782318,"status":"Unreachable"}

Answer 1

There is a configuration item heartbeat_expired_interval in the meta service. If the data node does not report a heartbeat after this time interval, the status will be set to Unreachable