Unity Catalog - External location AbfsRestOperationException

Question

I'am trying to setup connection between Databricks and Azure data lake storage gen2 using Unity Catalog External Locations feature.

Assumptions:

1. Adls is behind private endpoint

2. Databricks workspace is in private vnet, i've added Private and Public subnet of the workspace to ADLS account in "Firewalls and virtual networks" (service endpoint)

3. I've grant the ACL's to the service principal on container lvl of the storage account.

After creating service principal with Storage Blob Data Contributor role (i've also tried Storage Blob Data Owner, Storage Account Contributor and Contributor roles) and creating storage credentials with External Location associated with it, i got an error:

Error in SQL statement: UnityCatalogServiceException: [RequestId=6f9a0a07-513c-45a5-b2aa-a67dd7d7e662 ErrorClass=INVALID_STATE] Failed to access cloud storage: AbfsRestOperationException

on the other hand:

After creating mount connection using the same service prinicpal i am able to connect the storage and write/read data to it.

Do you have any ideas?

When i try connect to the Adls using Managed Identity with the "Access Connector" the problem is gone, but it is now in public preview:

https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/azure-managed-identities

Answer 1

I have the same issue. I did notice then when the storage account network firewall is disabled on the datalake it works using the service principle as the storage credential. I tried to add the public IP addresses from databricks found here but that did fail as well. Not sure how (from what IP address) to discover how Unity Catalog connects to the storage account. I have raised a support ticket with Microsoft and Databricks, will update once i hear more.

Answer 2

I fixed the issue by creating two Databricks connectors, one for accessing the metastore storage account and the other for accessing the data lake store account.