I am using Glue Studio 4.0 to choose data source (delta table 2.1.0 that saved in S3) as image below:
And then, I generate script from the box:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
S3bucket_node1 = glueContext.create_dynamic_frame.from_catalog(
database="tien_bronze_layer",
table_name="dim_product_dt",
transformation_ctx="S3bucket_node1",
)
job.commit()
Finally, I saved this job and run but I got an error:
Error: An error occurred while calling o96.getDynamicFrame. s3://tientest/Bronze_layer/dim_product_dt/_symlink_format_manifest/manifest is not a Parquet file. Expected magic number at tail, but found [117, 101, 116, 10]
I know this error but I don't find any docs to read delta tables that contain manifest. Can you all help me in this case, thanks!