how can I download selective date range files from S3 bucket based on given date range like 08th aug to 15 Aug using AWS CLI?

Question

I am able to filter a particular date's data but not the date range data. Like 12-09-2019 to 15-09-2019 using AWS CLI

eg. to filter 2019 year's data i am using --recursive --exclude "*" --include "2019"

Answer 1

You will need to use the s3api to handle the query which uses the JMESPath syntax

aws s3api list-objects-v2 --bucket BUCKET --query "Contents[?(LastModified>='2019-09-12' && LastModified<='2019-09-15')].Key"

You can also specify time as well

aws s3api list-objects-v2 --bucket BUCKET --query "Contents[?(LastModified>='2019-09-12T12:00:00.00Z' && LastModified<='2019-09-15T12:00:00.00Z')].Key"

The downside to this approach is that it must list every object and perform the query. For large buckets if you can limit to a prefix it will speed up your search.

aws s3api list-objects-v2 --bucket BUCKET --prefix PREFIX --query "Contents[?(LastModified>='2019-09-12T12:00:00.00Z' && LastModified<='2019-09-15T12:00:00.00Z')].Key"

And if your primary lookup is by date then look to store the objects in date/time sort order as you can use the prefix option to speed up your searches. A couple of examples.

prefix/20190615T041019Z.json.gz
2019/06/15/T041019Z.json.gz

This will