Name Node is going to Safe mode repeatedly

Question

We are using HDP 3.0 in my project which is in 2 node. For every 20-30 days name node is going to safe mode with below errors. For every time I am manually executing below command to leave safe mode.

hdfs dfsadmin -safemode leave

below are the logs of secondary namenode.(we lost namendoe logs)

ERROR namenode.SecondaryNameNode (SecondaryNameNode.java:doWork(365)) - Exception in doCheckpoint
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Log not rolled. Name node is in safe mode.
Resources are low on NN. Please add or free up more resourcesthen turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. NamenodeHostName

may i know the reason why namenode is going to safemode repetedly.

Answer 1

As the error message indicates, there is no disk space available in the node for namenode or hdfs to run. If you use ambari to manage your cluster you can go and view available space in each of your nodes by going to ambari dashboard and then click on Hosts and check space on each of the assigned datanodes or you can manually go to each of your datanodes and type df -hl to check whether the disk is full or support.

Solution: Add New Datanodes. Once added HDFS will redistribute the data.