Cloud formation template for delta lake tables using Aws glue crawler 'Invalid delta target with empty delta tables' error

Question

AWSTemplateFormatVersion: "2010-09-09"
    
AWSTemplateFormatVersion: "2010-09-09"

Description: Delta lake tables using aws glue crawler

Resources:
  Crawl:
    Type: AWS::Glue::Crawler
    Properties:
      Name: crawler_name
      DatabaseName: database_name
      Role: role_arn
      Targets:
        DeltaTargets:
          - DeltaTables: 
            - S3_path_for_delta_table_folder (having deltalogs as child inside)
          - WriteManifest: True 

I am expecting to create delta table with symlink with write manifest file enable. I am able to do with aws console but failing in cft with same data path

Answer 1

Please use the below CloudFormation to create a Crawler for Delta source to create the manifest.

I added the below property to enforce as non native Delta Table and tested it which works as expected :

CreateNativeDeltaTable : false

AWSTemplateFormatVersion: "2010-09-09"
Description: Delta lake tables using aws glue crawler
Resources:
  MyDeltaCrawler:
    Type: AWS::Glue::Crawler
    Properties:
      Name: "DeltaCrawler"
      Role: "arn:aws:iam::12345678910:role/GlueServiceRole"
      Tags: 
        "env": "dev1"
      DatabaseName: "default"
      Targets:
        DeltaTargets: 
          - CreateNativeDeltaTable: false
            DeltaTables: 
            - s3://your-bkt/path/to/delta/
            WriteManifest: true