Avoid pre-signed URL expiry when IAM role key rotates

Question

In Airflow I have 2 tasks defined that run every day:

• the first one creates a zip file and saves it in AWS under s3://{bucket-name}/foo/bar/{date}/archive.zip
• the second one pre-signs that url (should expire in 7 days) and sends it to Slack.

Because Qubole uses an IAM role the generated url will expire when the keys are rotated (less than 24 hours as far as I can tell).

I'm trying to find a solution for this. My current idea is moving the second task into an AWS lambda and using IAM user credentials to avoid the expiry issue.

Is there any other approach I could take without over complicating it?

Answer 1

You will need to use specific IAM credentials. Best practice would be be setting the permissions on the IAM user to only do what is required.