Hdfs balancer not working

Question

I'm sure nobody started the balancing, what could be wrong?

Balancing run thus: # hdfs balancer -threshold 1 16/02/19 15:10:02 INFO balancer.Balancer: Using a threshold of 1.0 16/02/19 15:10:02 INFO balancer.Balancer: namenodes = [hdfs://m1.h.c.test:8020, hdfs://m1.h.c.test] 16/02/19 15:10:02 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 1.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] 16/02/19 15:10:02 INFO balancer.Balancer: included nodes = [] 16/02/19 15:10:02 INFO balancer.Balancer: excluded nodes = [] 16/02/19 15:10:02 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved java.io.IOException: Another Balancer is running.. Exiting ...

I use hdp packages: ii hadoop-2-3-2-0-2950 2.7.1.2.3.2.0-2950
ii hadoop-2-3-2-0-2950-client 2.7.1.2.3.2.0-2950
ii hadoop-2-3-2-0-2950-hdfs 2.7.1.2.3.2.0-2950
ii hadoop-2-3-2-0-2950-hdfs-namenode 2.7.1.2.3.2.0-2950

Answer 1

It can also be problem described in https://issues.apache.org/jira/browse/HDFS-8897 - this is specific to HDFS HA configuration.

Issue fixed in HDFS 2.8 and 3.0.

Answer 2

You can try: hdfs dfs -rm -skipTrash /system/balancer.id

Answer 3

Just run the following command to kill the process.

kill $(ps aux | grep 'balancer' | awk '{print $2}')

And then run balancer using Ambari.

In case you are using HA environment, first find out where the process is running by using,

hdfs dfs -cat /system/balancer.id