AWS Glue Job Monthly Report

66 Views Asked by At

I have an AWS Glue job that targets to generate a monthly report

I am using aws s3 parquet format as my source and validating query at athena

current issue is when my aws glue job runs the following day it only aggregated 1 day only

adv_amt |adv_fee|adv_txn|adv_uniq_user|credit_date
300.0   |60.0   |6      |2            | 2023-05-01
260.0   |52.0   |6      |3            | 2023-04-01
170.0   |34.0   |5      |3            | 2023-03-01

my target output scenario is i want to run it daily and update current month.

eg. today is 2023-05-19

  1. my current result is aggregated from 2023-05-01 to 2023-05-19 as 2023-05-01
  2. if i run the job on the following day it should be aggregated from 2023-05-01 to 2023-05-20 as 2023-05-01

any idea if there is a way to get my expected result?

0

There are 0 best solutions below