Getting an error while using copy activity (polybase) in adf to copy parquet files in ADLS gen2 to Azure synapse table

Question

My source is parquet files in ADLS gen2. All the parquet files are part files of size 10-14 MB. The total size should be around 80 GB

Sink is Azuresynapse table.

Copy method is Polybase. Getting below error within 5 sec of execution like below:

ErrorCode=PolybaseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse. Operation: 'Create external table'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, URL',Source=.Net SqlClient Data Provider,SqlErrorNumber=105019,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=105019,State=1,Message=External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD,

Answer 1

I've seen this error due to failed authentication, check whether the authorization header and/or signature is wrong. For example, create the scope credential using your ADLS Gen2 storage account access key:

CREATE DATABASE SCOPED CREDENTIAL [MyADLSGen2Cred] WITH
  IDENTITY='user',
  SECRET='zge . . . 8V/rw=='

The external data source is created as follows:

CREATE EXTERNAL DATA SOURCE [MyADLSGen2] WITH (
  TYPE=HADOOP,
  LOCATION='abfs://[email protected]',
  CREDENTIAL=[MyADLSGen2Cred])

You can specify wasb instead of abfs, and if you're using SSL, specify it as abfss. Then the external table is created as follows:

CREATE EXTERNAL TABLE [dbo].[ADLSGen2] (
  [Content] varchar(128))
WITH (
  LOCATION='/',
  DATA_SOURCE=[MyADLSGen2],
  FILE_FORMAT=[TextFileFormat])

You can find additional information in my book "Hands-On Data Virtualization with Polybase".