Databricks connection to Azure SQL Database with ActiveDirectoryPassword

Question

In my Python 3 notebook in Azure Databricks, when I run this:

%scala
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._

val config = Config(Map(
  "url"            -> "serverName.database.windows.net:1433",
  "databasename"   -> "dbName",
  "user"           -> "[email protected]",
  "password"       -> "password",
  "encrypt"        -> "true",
  "trustServerCertificate" -> "false",
  "hostNameInCertificate"  -> "*.database.windows.net",
  "loginTimeout"   -> "30",
  "authentication" -> "ActiveDirectoryPassword",
  "dbTable"        -> "dbo.TableName"
))

val collection = sqlContext.read.sqlDB(config)
collection.show()

I get the error:

java.lang.NoClassDefFoundError: com/microsoft/aad/adal4j/AuthenticationException

This database requires ActiveDirectoryPassword. I am able to connect using pyodbc on my computer with the credentials above, but I can't get anything to connect from Databricks. This is an Azure Databricks Standard account (not Premium). Any ideas?

UPDATE: Thanks Mark for the answer. Apparently, the default importing of jar's in Azure Databricks puts them in the application class path instead of the system class path, which is the cause of this error (according to: https://forums.databricks.com/questions/706/how-can-i-attach-a-jar-library-to-the-cluster-that.html). To resolve this, I used the code below (change 'clusterName' to actual name of cluster):

%scala
// This code block only needs to be run once to create the init script for the cluster (file remains on restart)

// Create dbfs:/databricks/init/ if it doesn’t exist.
dbutils.fs.mkdirs("dbfs:/databricks/init/")

// Display the list of existing global init scripts.
display(dbutils.fs.ls("dbfs:/databricks/init/"))

// Create a directory named (clusterName) using Databricks File System - DBFS.
dbutils.fs.mkdirs("dbfs:/databricks/init/clusterName/")

// Create the adal4j script.
dbutils.fs.put("/databricks/init/clusterName/adal4j-install.sh","""
#!/bin/bash
wget --quiet -O /mnt/driver-daemon/jars/adal4j-1.6.0.jar http://central.maven.org/maven2/com/microsoft/azure/adal4j/1.6.0/adal4j-1.6.0.jar
wget --quiet -O /mnt/jars/driver-daemon/adal4j-1.6.0.jar http://central.maven.org/maven2/com/microsoft/azure/adal4j/1.6.0/adal4j-1.6.0.jar""", true)


// Check that the cluster-specific init script exists.
display(dbutils.fs.ls("dbfs:/databricks/init/clusterName/"))