Using Azure SQLDW Polybase for data ingestion from ADLS Gen 1 using vnet service endpoints

184 Views Asked by At

I am trying to use Polybase in Azure SQL Data Warehouse (SQLDW) to ingest data (persisted in Parquet format from a Hadoop cluster in a VNET) on Azure Data Lake Store (ADLS) Gen 1. The process is working fine but the throughput I am getting is quite poor i.e. approximately 10MBps. My assumption is that the traffic is going via Internet and not via Azure backbone network. To address this, I've enabled VNET service end-point as follows: VNET to ADLS (as per this link) VNET to Azure SQL Data Warehouse (as per this link)

However, even after doing so, there is no performance gain. My understanding is that after enabling this, the traffic should go through Azure backbone network but there is no difference. Am I missing anything in this whole workflow?

0

There are 0 best solutions below