Create dynamic frame from S3 bucket AWS Glue

Question

Summary: I've got a S3 bucket which contains list of JSON files. Bucket contains child folders which are created by date. All the files contain similar file structure. Files get added on daily basis.

JSON Schema

schema = StructType([
    StructField("main_data",StructType([
        StructField("action",StringType()),
        StructField("parameters",StructType([
            StructField("project_id",StringType()),
            StructField("integration_id",StringType()),
            StructField("cohort_name",StringType()),
            StructField("cohort_id",StringType()),
            StructField("cohort_description",StringType()),
            StructField("session_id",StringType()),
            StructField("users",StructType([StructField("user_id",StringType())]))
        ]),
    )]
    )),
    StructField("lambda_data", StructType([
            StructField("date",LongType())
        ]))
    ]) 

Question I am trying to create dynamic frame from options where source is S3 and type is JSON. I'm using following code however it is not returning any value. Where am I going wrong?

Script

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from functools import reduce
from awsglue.dynamicframe import DynamicFrame

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)


    df = glueContext.create_dynamic_frame.from_options(
        connection_type = 's3',
        connection_options={'paths':['Location for S3 folder']},
        format='json',
        # formatOptions=$..*
        )
        
    
    print('Total Count:')
    df.count()

Answer 1

Can you check if your Glue Role has access to the S3 bucket,

Also in connection_options add

"recurse" : True