Hadoop error du: java.util.ConcurrentModificationException

Question

While working on my HDFS cluster, I get this error

du: java.util.ConcurrentModificationException

whenever I run

hdfs dfs -du -h -s /some/path/

A quick check on the Internet and I saw it was bug in Hadoop 2.7.0.

Answer 1

To fix the issue, I had to delete some of my Hadoop snapshot files. I believe a/some snapshot(s) had been corrupted as I had one of my data node decommissioned uncleanly from my cluster few days ago.

hdfs lsSnapshottableDir
drwxr-xr-x 0 hdfs supergroup 0 2018-01-30 17:04 0 65536 /data

[hdfs@hmastera ~]$ hdfs dfs -ls /data/.snapshot
Found 5 items
drwxr-xr-x   - hdfs supergroup          0 2017-08-19 01:06 /data/.snapshot/insight-dl-cluster_snapshot_20170819T010503
drwxr-xr-x   - hdfs supergroup          0 2017-08-19 01:08 /data/.snapshot/insight-dl-cluster_snapshot_20170819T010746
drwxr-xr-x   - hdfs supergroup          0 2017-08-19 01:12 /data/.snapshot/insight-dl-cluster_snapshot_20170819T011013
drwxr-xr-x   - hdfs supergroup          0 2017-08-19 01:14 /data/.snapshot/insight-dl-cluster_snapshot_20170819T011219
drwxr-xr-x   - hdfs supergroup          0 2018-01-13 16:24 /data/.snapshot/insight-dl-cluster_snapshot_20180113T162234

`

Then I started deleting the snapshots till I got my mojo back.

   hdfs@ hmastera ~]
        hdfs dfs -deleteSnapshot /data  insight-dl-cluster_snapshot_20170819T010503
        hdfs dfs -deleteSnapshot /data  insight-dl-cluster_snapshot_20170819T010746
        hdfs dfs -deleteSnapshot /data  insight-dl-cluster_snapshot_20170819T011013
        hdfs dfs -deleteSnapshot /data  insight-dl-cluster_snapshot_20170819T011219
        hdfs dfs -deleteSnapshot /data insight-dl-cluster_snapshot_20180113T162234

[hdfs@ hmastera ~]$ hdfs dfs -du -h -s /data
        510.1 G  /data