python (3.11) lambda function using stored credentials

39 Views Asked by At

First, I've spent a lot of time reading through various previous iterations of this question with no joy so far.

I've got a python3.11 lambda running in a central account in an AWS org that:

  1. Is triggered by an SNS topic supplying events containing details of each unique account-region combo in the org
  2. Uses sts to assume a role in that account

    try:
        client = boto3.client("sts")
        r = client.assume_role(
            RoleArn=f"arn:aws:iam::{account_id}:role/{role}",
            RoleSessionName=role_session_name,
        )
  1. creates session creds

   try:
        session = boto3.session.Session(
            aws_access_key_id=r["Credentials"]["AccessKeyId"],
            aws_secret_access_key=r["Credentials"]["SecretAccessKey"],
            aws_session_token=r["Credentials"]["SessionToken"],
            region_name=region,
        )
  1. Does stuff in the account...

What I'm seeing though is that it fails roughly 20-30% of the time with Exception Occurred Accessing AWS Config API: The security token included in the request is invalid

Across different invocations an account-region combo that fails in one, works fine in another, so the permissions are fine. The issue (as far as I can tell) appears to be the way it's accessing it's credentials. My suspicion is that the lambda is re-using credentials from a previous invocation. It will often fail within the same account, just a different region.

The role assumption and session creation blocks are inside try-except blocks and have no issues. The lambda bombs out when it actually attempts to access resources.

I've tried limiting concurrency to 1. No joy. I've added logger.debug statements everywhere and confirmed that the session creds are being generated correctly.

I'm wondering if there's a way of ensuring each lambda invocation runs in a discrete environment?

0

There are 0 best solutions below