Read a nested json Map in StepFunction parameter

569 Views Asked by At

I am trying to read a static map's value based on a dynamic key in step function. In the example down below I am trying to access timestamps from a static map StartTimeOverrides given a bunch of dynamic city names from a lambda. I am facing challenges in fetching these timestamps, the details of the problems are mentioned after the sample step function -

{
  "StartAt": "GetCities",
  "States": {
    "GetCities": {
      "Next": "CityIterator",
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Type": "Task",
      "OutputPath": "$.Payload",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:us-west-2:xxxx:function:GetCities",
        "Payload.$": "$"
      }
    },
    "CityIterator": {
      "Type": "Map",
      "ResultPath": null,
      "Next": "Workflow finished successfully.",
      "Parameters": {
        "City.$": "$$.Map.Item.Value",
        "StartTimeOverrides": {
          "NewYork": "22:30:00",
          "Seattle": "11:00:00",
          "Chicago": "10:00:00"
        }
      },
      "Iterator": {
        "StartAt": "GetTimestamp",
        "States": {
          "GetTimestamp": {
            "Type": "Pass",
            "Next": "ShouldDelayAlgoRun",
            "Parameters": {
              "DelayAlgoUntil.$": "$.StartTimeOverrides.NewYork", // Want to make this something like $.StartTimeOverrides.$.City
              "City.$": "$.City"
            },
            "ResultPath": "$"
          },
          "ShouldDelayAlgoRun": {
            "Type": "Choice",
            "Choices": [
              {
                "Variable": "$.DelayAlgoUntil",
                "IsTimestamp": true,
                "Next": "DelayAlgoRun"
              }
            ],
            "Default": "RunAlgo"
          },
          "RunAlgo": {
            "End": true,
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Catch": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "Next": "HandleFailure"
              }
            ],
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "arn:aws:lambda:us-west-2:xxxx:function:RunAlgo",
              "Payload": {
                "CityName.$": "$.City"
              }
            }
          },
          "DelayAlgoRun": {
            "Type": "Wait",
            "TimestampPath": "$.DelayAlgoUntil",
            "Next": "RunAlgo"
          },
          "HandleFailure": {
            "Type": "Pass",
            "End": true
          }
        }
      },
      "ItemsPath": "$.City",
      "MaxConcurrency": 5
    },
    "Workflow finished successfully.": {
      "Type": "Succeed"
    }
  }
}

Attaching image to help visualize the stepFunction.Workflow visualization

Sample Input to GetTimestamp step -

{
  "StartTimeOverrides": {
    "NewYork": "22:30:00",
    "Seattle": "11:00:00",
    "Chicago": "10:00:00"
  }
  "City": "NewYork"
}

Sample Output from GetTimestamp step -

{
  "City": "NewYork",
  "DelayAlgoUntil: "22:30:00"
}

For other cities this will not give the right DelayAlgoUntil value since we have hardcoded the DelayAlgoUntil for NewYork in our state machine definition, is there a way to make this dynamic?

When I try $.StartTimeOverrides.$.City it gives me an error - An error occurred while executing the state 'GetTimestamp' (entered at the event id #12). The JSONPath '$.StartTimeOverrides.$.City' specified for the field 'DelayAlgoUntil.$' could not be found in the input - {..json map..}

Any help or guidance appreciated. Thanks

Additional Context: What am I trying to do in this step function. I have a lambda which fetches a bunch of cities dynamically from a source. For each city we have timestamps at which some algorithm should run daily. If the city is not present in the static map having timestamps then the Algo is supposed to be run immediately at the time of step function execution.

0

There are 0 best solutions below