ADAL4J javax mail parse exception when trying to connect to Azure SQL DB from Spark Submit

Question

I am trying to use the Spark connector for Azure SQL Databases library (GitHub) to get data from Spark and bulk insert it into an Azure Database. I am trying to use 'ActiveDirectoryPassword' as an authentication method, but adal4j always throws an AuthenticationException exception that states a javax.mail.internet.ParseException was thrown.

I am building a fat jar with all libraries included and running it with spark-submit.

The Azure Organization has a custom domain.

For the user config option I have tried:

• [email protected]
• myUser
• [email protected]@myDB.database.windows.net
• [email protected]@database.windows.net
• [email protected]

When I leave the field empty, it throws an error that the username is empty.

I can use that user to log into Azure Data Studio via Azure Active Directory.

Here is the snipped I use to load the data:

val config = Config(Map(
    "url" -> "myDB.database.windows.net",
    "databaseName" -> "DatabaseName",
    "user" -> "[email protected]",
    "password" -> "****",
    "authentication" -> "ActiveDirectoryPassword",
    "encrypt" -> "true",
    "trustServerCertificate" -> "false",
    "hostNameInCertificate" -> "myDB.database.windows.net",
    "dbTable" -> "test.testing",
    "bulkCopyBatchSize" -> "2500",
    "bulkCopyTableLock" -> "true",
    "bulkCopyTimeout" -> "600"
  ))

  val session = SparkSession.builder.appName("App").enableHiveSupport.getOrCreate
  val sqlContext = session.sqlContext
  val df = sqlContext.sql(s"SELECT * FROM myTable")

  df.bulkCopyToSqlDB(config)

and here the exception that is thrown:

com.microsoft.sqlserver.jdbc.SQLServerException: Failed to authenticate the user [email protected] in Active Directory (Authentication=ActiveDirectoryPassword).
        at com.microsoft.sqlserver.jdbc.SQLServerADAL4JUtils.getSqlFedAuthToken(SQLServerADAL4JUtils.java:62)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.getFedAuthToken(SQLServerConnection.java:4264)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.onFedAuthInfo(SQLServerConnection.java:4237)
        ......Truncated for readability......
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.concurrent.ExecutionException: com.microsoft.aad.adal4j.AuthenticationException: javax/mail/internet/ParseException
        at com.microsoft.sqlserver.jdbc.SQLServerADAL4JUtils.getSqlFedAuthToken(SQLServerADAL4JUtils.java:60)
        ... 48 more
Caused by: com.microsoft.aad.adal4j.AuthenticationException: javax/mail/internet/ParseException
        at com.microsoft.sqlserver.jdbc.SQLServerADAL4JUtils.getSqlFedAuthToken(SQLServerADAL4JUtils.java:53)
        ... 48 more

Thanks in advance

Answer 1

Override the javax.mail jar with latest version → javax.mail-1.6.2.jar using 'spark.driver.extraClassPath' and 'spark.executor.extraClassPath' options in your spark-submit command and run it, should work.