Connecting to Kerberrized HDFS , java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name;

Question

I am trying to connect to a Kerberized hdfs cluster , with below code , with same below code i am able to access hbase ofcourse with HBaseConfiguration,

Configuration config = new Configuration();
config.set("hadoop.security.authentication", "Kerberos");

UserGroupInformation.setConfiguration(config);
UserGroupInformation ugi = null;
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("me@EXAMPLE>COM","me.keytab");
model = ugi.doAs((PrivilegedExceptionAction<Map<String,Object>>) () -> { 
  testHadoop(hcb.gethDFSConfigBean());
  return null;
});

I have been able to successfully access Solr, Impala with same keytab and principal, I am getting this weird Failed to find service name for hdfs.

Please look at below stack trace

java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "Securonix-int3.local/10.0.4.36"; destination host is: "sobd189.securonix.com":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil.lambda$main$4(SnyperUIUtil.java:1226)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil$$Lambda$6/1620890840.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil.main(SnyperUIUtil.java:1216)
Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    at org.apache.hadoop.ipc.Client.call(Client.java:1438)
    ... 23 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
    at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:322)
    at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:231)
    at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
    at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)

After i enabled the debug code for Kerberos i got the below debug log when i call FileSystem.get(); Kerberor debug log :

Java config name: null Java config name: null Native config name: /etc/krb5.conf Native config name: /etc/krb5.conf Loaded from native config Loaded from native config 16/02/22 15:53:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Java config name: null Java config name: null Native config name: /etc/krb5.conf Native config name: /etc/krb5.conf Loaded from native config Loaded from native config

KdcAccessibility: reset >>> KdcAccessibility: reset KdcAccessibility: reset >>> KdcAccessibility: reset KeyTabInputStream, readName(): EXAMPLE.COM >>> KeyTabInputStream, readName(): EXAMPLE.COM KeyTabInputStream, readName(): securonix >>> KeyTabInputStream, readName(): securonix KeyTab: load() entry length: 55; type: 23 >>> KeyTab: load() entry length: 55; type: 23 KeyTabInputStream, readName(): EXAMPLE.COM >>> KeyTabInputStream, readName(): EXAMPLE.COM KeyTabInputStream, readName(): securonix >>> KeyTabInputStream, readName(): securonix KeyTab: load() entry length: 71; type: 18 >>> KeyTab: load() entry length: 71; type: 18 Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 default etypes for default_tkt_enctypes: 18 18 16. default etypes for default_tkt_enctypes: 18 18 16. KrbAsReq creating message >>> KrbAsReq creating message KrbKdcReq send: kdc=sobd189.securonix.com TCP:88, timeout=30000, number of retries =3, #bytes=139 >>> KrbKdcReq send: kdc=sobd189.securonix.com TCP:88, timeout=30000, number of retries =3, #bytes=139 KDCCommunication: kdc=sobd189.securonix.com TCP:88, timeout=30000,Attempt =1, #bytes=139 >>> KDCCommunication: kdc=sobd189.securonix.com TCP:88, timeout=30000,Attempt =1, #bytes=139 DEBUG: TCPClient reading 639 bytes >>>DEBUG: TCPClient reading 639 bytes KrbKdcReq send: #bytes read=639 >>> KrbKdcReq send: #bytes read=639 KdcAccessibility: remove sobd189.securonix.com >>> KdcAccessibility: remove sobd189.securonix.com Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType KrbAsRep cons in KrbAsReq.getReply securonix

Interestingly when i use api of the file system like hdfs.exists()

 >>>KinitOptions cache name is /tmp/krb5cc_501
 >> Acquire default native Credentials
 default etypes for default_tkt_enctypes: 18 18 16.
 >>> Found no TGT's in LSA

Answer 1

I had the same issue with Spark2 and HDP3.1, using Isilon/OneFS as storage instead of HDFS.

The OneFS service management pack doesn't provide configuration for some of the HDFS parameters that are expected by Spark2 (they aren't available at all in Ambari), such as dfs.datanode.kerberos.principal. Without these parameters Spark2 HistoryServer may fail to start and report errors such as "Failed to specify server's principal name".

I added the following properties to OneFS under Custom hdfs-site:

dfs.datanode.kerberos.principal=hdfs/_HOST@<MY REALM>
dfs.datanode.keytab.file=/etc/security/keytabs/hdfs.service.keytab
dfs.namenode.kerberos.principal=hdfs/_HOST@<MY REALM>
dfs.namenode.keytab.file=/etc/security/keytabs/hdfs.service.keytab 

This resolved the initial error. Thereafter I was getting an error of the following form:

Server has invalid Kerberos principal: hdfs/<isilon>[email protected], expecting: hdfs/[email protected]

This was related to cross-realm authentication. Resolved by adding the below setting to custom hdfs-site:

dfs.namenode.kerberos.principal.pattern=*

Answer 2

I believe the problem is that HDFS expects the Configuration to have a value for dfs.datanode.kerberos.principal, which is the principal of the datanodes, and it is missing in this case.

I had this same problem when I created a Configuration instance from only core-site.xml and forgot to add hdfs-site.xml. As soon as I added hdfs-site.xml it started working, and hdfs-site.xml had:

 <property>
      <name>dfs.datanode.kerberos.principal</name>
      <value>....</value>
 </property>

Hope this helps.