assumeRole in airflow container

Question

I'm trying to run local docker containers as airflow cluster for local testing. My DAG needs to run some code like sts_client.assume_role. It works with ECS. However in local containers, it only gets my identity as arn:aws:sts::xxx:assumed-role/okta-dev/xxxx.com, which is not listed in the trust relationship of the role DAG code to assume. Questions are:

1. I think we can try to add the arn:aws:sts::xxx:assumed-role/okta-dev/xxxx.com to the trust relationship of assume role, but is there a better way to make it work? In ECS, the identity is like arn:aws:sts::xxx:assumed-role/xxx-container/xxx which has been added to target roles.
2. If we have to add the okta arn to the role, each of the team members needs to do that. In that case, can we make all okta arn to assume one role in, say, <middle_role>, and use add that middle role to target role DAG needs to assume? That is: Add all okta arns to trust relationship of middle_role => add middle_role to trust relationship of target roles.

Thanks

Answer 1

This might be more a question for an IAM/AWS expert but from the Airflow side there is a way to mount the local credentials in local containers by using a docker-compose.override.yml.

So you if you can assume the role you need (in your case the arn:aws:sts::xxx:assumed-role/xxx-container/xxx role) with the aws cli locally to have the correct credentials in .aws you can then you mount your credentials in docker-compose.override.yml (this is for Mac, so you might need to adjust the path to .aws):

version: "3.1"
services:
    scheduler:
        volumes:
        - /Users/<username>/.aws:/usr/local/airflow/.aws:ro
    webserver:
        volumes:
        - /Users/<username>/.aws:/usr/local/airflow/.aws:ro
    triggerer:
        volumes:
        - /Users/<username>/.aws:/usr/local/airflow/.aws:ro

and set the following ENV variables in your Dockerfile:

ENV AWS_CONFIG_FILE=/usr/local/airflow/.aws/config
ENV AWS_SHARED_CREDENTIALS_FILE=/usr/local/airflow/.aws/credentials

When you run Airflow, all AWS connections without defined credentials automatically fall back to these user credentials when connecting to AWS.

Might be worth trying. :)