Scenario: I have a hub & spoke architecture with Azure Firewall, which acts as my DNS server to VNets in all spokes. I also have a VPN connection, which I use to transfer data to my Azure Data Lake Storage Gen2. For that I use Private Endpoint, which is configured with Private DNS Zone associated with a Hub network. So for - all great, all my traffic is secure, wherever I connect to my ADLS Gen2 from premises or from Databricks in Azure. Now here's the problem. It costs a lot of money since Azure Private Endpoints charges you for inbound and outbound traffic.
So the question is, is there away to tell my Databricks nodes to use Service Endpoint (which is free) rather than Private Endpoint since Azure Firewall DNS always returns a private IP for my ADLS? I still need to keep Private Endpoint to be able to securely connect from my premises.
• Though you cannot create a service endpoint directly for the Azure Databricks workspace to transfer data from the Azure Data Lake Storage Gen 2 to it, but you can surely connect your Azure Databricks workspace to your on-premises network through the transit virtual network gateway created in the virtual network where your Azure Databricks workspace is peered. For that purpose, you will have to set up Azure virtual network gateway in the virtual network in which ADLS Gen2 has been deployed.
• Once the above said has been done, peer the virtual networks in which the private endpoint is configured and the one where ADLS Gen2 storage account is configured such that the Azure Databricks workspace is able to create a virtual network peering with the virtual network where Service endpoint for the Microsoft storage account is configured.
• Then configure the user defined routes and associate them with your Azure Databricks virtual network subnets and validate the setup. Kindly refer to the network diagram below for more clarification: -
For more information, kindly refer to the documentation link below: -
https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/on-prem-network