How to use jceks file stored in hdfs to connect to aws sqs queue and s3

Question

I have stored secret key and access key in a hdfs as a file, which are used to access AWS,

hadoop credential create fs.s3a.access.key -provider jceks://hdfs/user/dev/keys.jceks -value ****************

hadoop credential create fs.s3a.secret.key -provider jceks://hdfs/user/dev/keys.jceks -value **********

I want to use jceks file to connect to SQS queue and S3 from Java code.

Answer 1

I was able to solve this using below Code :

Java Code :

Configuration hadoopConfiguration = SparkSession.sparkContext().hadoopConfiguration();
log.info("CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH : "+hadoopConfiguration.get(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH ));
String accessKey = new String(hadoopConfiguration.getPassword("fs.s3a.access.key"));
String secretKey = new String(hadoopConfiguration.getPassword("fs.s3a.secret.key"));

Scala Code :

val hadoopConfiguration = sparkSession.sparkContext.hadoopConfiguration
hadoopConfiguration.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH, keyFileHdfsPath);
val access_Key = hadoopConfiguration.getPassword("fs.s3a.access.key").mkString
val secret_Key = hadoopConfiguration.getPassword("fs.s3a.secret.key").mkString