I have an AWS Glue job that targets to generate a monthly report
I am using aws s3 parquet format as my source and validating query at athena
current issue is when my aws glue job runs the following day it only aggregated 1 day only
adv_amt |adv_fee|adv_txn|adv_uniq_user|credit_date
300.0 |60.0 |6 |2 | 2023-05-01
260.0 |52.0 |6 |3 | 2023-04-01
170.0 |34.0 |5 |3 | 2023-03-01
my target output scenario is i want to run it daily and update current month.
eg. today is 2023-05-19
- my current result is aggregated from 2023-05-01 to 2023-05-19 as 2023-05-01
- if i run the job on the following day it should be aggregated from 2023-05-01 to 2023-05-20 as 2023-05-01
any idea if there is a way to get my expected result?