Passing inputs to lambda function inside Map state of step function

Question

I am trying to pass some variables for a lambda function event inside a map state. But somehow, The lambda inside the map state is only getting 5 default json key value pairs as input as below.

{
"Etag":"etag",
"Key":"s3objectURI",
"LastModified":1.69321E9,
"Size":500,
"StorageClass":"STANDARD"
}

The step function definition is as follows.

{
  "Comment": "A description of my state machine",
  "StartAt": "Lambda Invoke",
  "States": {
    "Lambda Invoke": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "FunctionName": "LambdaFunctionName",
        "Payload": {
          "bucket": "s3bucket",
          "prefix": "etl-aws/process/parameters/wf.json"
        }
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException",
            "Lambda.TooManyRequestsException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "Map"
    },
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "Lambda change content",
        "States": {
          "Lambda change content": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "OutputPath": "$.Payload",
            "Parameters": {
              "FunctionName": "AnotherLambdaFunctionInsideMapState-toTakeInputFromPreviousStates-HereTakingS3BucketName",
              "Payload": {
                "bucket.$": "$.s3_bucket_name",
                "Key.$": "$.Key"
              }
            },
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "End": true
          }
        }
      },
      "Label": "Map",
      "MaxConcurrency": 25,
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {
          "Bucket": "s3Bucket",
          "Prefix": "examplePrefixForParallelExecutions"
        }
      },
      "ResultPath": null,
      "End": true
    }
  }
}

This is a simple step function which contains a first lambda which returns some output keyvalue pairs. Next a map state contains a lambda which needs to use one of those key value pairs of first lambda output. When I am trying to run this, I am getting an error for key $.s3_bucket_name not avaialable as input for inside lambda function. I could see the bucket key available in map state input. Could anyone please let me know how I can pass the keys to inside lambda function. Let me know if anything is confusing and needs clarity. Thanks in advance.

Answer 1

Your Distributed Map state includes an ItemReader, which tells your Step Functions Workflow with Resource of arn:aws:states:::s3:listObjectsV2 which tells Step Functions to iterate over that bucket and to use the meta-data for selected items as input to the child workflow executions.

If that is not what you want and your first Lambda function returns an array that you want to use as the set of Items for your Map state, then you should remove the ItemReader.

If you do want to iterate over these Items from S3 but you also want to make the output from the first Lambda function available for each of the child workflow executions, you need to use ItemSelector. It will allow you to shape the input to each child workflow execution such that it can combine elements of input to the overall Map state with input sourced from S3 by the ItemReader.

The simplest example to illustrate this would be an ItemSelector like the following.

"ItemSelector": {
   "item.$": "$$.Map.Item.Value",
   "state_input.$": "$"
}

With this, if the output from your first Lambda function which is the input to the Map state was:

{
  "s3_bucket_name": "mybucketname",
  "somethingelse": "somevalue"
}

Then your child workflow executions would get something like this as input:

{
  "item": {
    "Etag": "\"fb895f3dfeb060a0b34c28747b623d11\"",
    "Key": "folder1/folder2/file.csv",
    "LastModified": 1690555308,
    "Size": 20919,
    "StorageClass": "STANDARD"
  },
  "state_input": {
    "s3_bucket_name": "mybucketname",
    "somethingelse": "somevalue"
  }
}