I am using Glue Studio 4.0 to choose data source (delta table 2.1.0 that saved in S3) as image below:

enter image description here

And then, I generate script from the box:


    import sys
    from awsglue.transforms import *
    from awsglue.utils import getResolvedOptions
    from pyspark.context import SparkContext
    from awsglue.context import GlueContext
    from awsglue.job import Job
        
    args = getResolvedOptions(sys.argv, ["JOB_NAME"])
    sc = SparkContext()
    glueContext = GlueContext(sc)
    spark = glueContext.spark_session
    job = Job(glueContext)
    job.init(args["JOB_NAME"], args)
    S3bucket_node1 = glueContext.create_dynamic_frame.from_catalog(
                database="tien_bronze_layer",
                table_name="dim_product_dt",
                transformation_ctx="S3bucket_node1",
            )
    job.commit()

Finally, I saved this job and run but I got an error:

 
Error: An error occurred while calling o96.getDynamicFrame. 
s3://tientest/Bronze_layer/dim_product_dt/_symlink_format_manifest/manifest is not a Parquet file. Expected magic number at tail, but found [117, 101, 116, 10]

I know this error but I don't find any docs to read delta tables that contain manifest. Can you all help me in this case, thanks!

0

There are 0 best solutions below