httpfs error Operation category READ is not supported in state standby

Question

I am working on hadoop apache 2.7.1 and I have a cluster that consists of 3 nodes

nn1
nn2
dn1

nn1 is the dfs.default.name, so it is the master name node.

I have installed httpfs and started it of course after restarting all the services. When nn1 is active and nn2 is standby I can send this request

http://nn1:14000/webhdfs/v1/aloosh/oula.txt?op=open&user.name=root

from my browser and a dialog of open or save for this file appears, but when I kill the name node running on nn1 and start it again as normal then because of high availability nn1 becomes standby and nn2 becomes active.

So here httpfs should work, even if nn1 becomes stand by, but sending the same request now

http://nn1:14000/webhdfs/v1/aloosh/oula.txt?op=open&user.name=root

gives me the error

{"RemoteException":{"message":"Operation category READ is not supported in state standby","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}

Shouldn't httpfs overcome nn1 standby status and bring the file? Is that because of a wrong configuration, or is there any other reason?

My core-site.xml is

<property>
       <name>hadoop.proxyuser.root.hosts</name>
                <value>*</value>
       </property>

        <property>
                <name>hadoop.proxyuser.root.groups</name>
                <value>*</value>
        </property>

Answer 1

It looks like HttpFs is not High Availability aware yet. This could be due to the missing configurations required for the Clients to connect with the current Active Namenode.

Ensure the fs.defaultFS property in core-site.xml is configured with the correct nameservice ID.

If you have the below in hdfs-site.xml

<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>

then in core-site.xml, it should be

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://mycluster</value>
</property>

Also configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the currently Active and is serving client requests.

Add this property to hdfs-site.xml

<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>            
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

Restart the Namenodes and HttpFs after adding the properties in all nodes.

Answer 2

Not sure if it is solving this issue, but solved mine when I was accessing HDFS via HDFS client and it was constantly logging warning about reading in standby mode. I changed following property in hdfs-site.xml for HDFS client:

  <property>
    <name>dfs.client.failover.proxy.provider.uniza-hdfs-ha</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>

to

  <property>
    <name>dfs.client.failover.proxy.provider.uniza-hdfs-ha</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider</value>
  </property>

It is also mentioned in official documentation https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html