AWS Glue catalogPartitionPredicate : to_date is not working

Question

I am planning to utilize catalogPartitionPredicate in one of my projects. I am unable to handle one of the scenarios. Below are the details:

1. Partition columns: Year,Month & Day
2. catalogPartitionPredicate: year>='2021' and month>='12'

If the year changes to 2022(2022-01-01) and I want to read data from 2021-12-01; the expression won't be able to handle as it will not allow to read 2022 data. I tried to concat the partition keys but it didn't work.

Is there any way to implement to_date functionality or any other workaround to handle this scenario?

Answer 1

You don't need the month, use only the "year>='2021'". It is the same logic you are using regarding the "day"