Generating own AWS Trace Header on an SQS message does not prevent Recursive Loop on a Lambda

Question

I'm designing a system solution where an SQS queue triggers a Lambda, the queue receives a lot of messages for a logical grouping say an execId. The lambda reads the messages from the queue in a batch and for the same execId, it creates a new message adding a firstMsgTimestamp and lastMsgTimestamp of the messages received from the queue and puts it back into the same queue. For messages with the firstMsgTimestamp and lastMsgTimestamp, it checks whether the lastMsgTimestamp was more than 2 minutes ago, if it is, instead of publishing the message back to the queue, it handles downstream events (like making an API call, or invoking another lambda, etc) - this ensures that no message that is put back into the queue runs recursively after 2 minutes. The architecture would be the one attached below.

However, AWS reports that this lambda is running recursively and sends the following alert in the AWS Health Dashboard

AWS Lambda has detected that one or more Lambda functions in your AWS Account are being invoked in a recursive loop with other AWS resources.

Upon research, I've found that SQS uses XRay Trace Headers to identify recursive loops as mentioned in the AWS post here here.

I generated the AWS Tracing Header uniquely for each message and sent the message to the SQS where each message's content would be unique with the following piece of code:

{
  Id: crypto.randomUUID(),
  MessageBody: JSON.stringify(message),
  MessageSystemAttributes: {
    AWSTraceHeader: {
      DataType: 'String',
      StringValue: generateTraceHeader(),
    },
  },
}

I added logs to verify that the header generated by the generateTraceHeader() function is the same header that gets into the message by logging the entire message record as well and they both are the same so I'm sure the message has the AWS Tracing Header that I've generated. Additionally, the AWS Trace Header I generated doesn't have the Lineage Header info which AWS themselves inject into the header which also confirms that the messages have the Tracing Header that I've generated.

However, AWS still reports Recursive loops for the lambda and drops the lambda invocations (screenshot attached below):

I also tried disabling tracing for the SQS or the lambda, but it looks like Node SDK doesn't support disabling tracing at the moment as there's an open GitHub issue here.

Does anyone know how I could prevent AWS from detecting this as a recursive loop without having to contact AWS and asking them to turn off recursive loop prevention for Lambda?

Answer 1

I had a similar issue. I fixed it by these steps:

• set and fix the version of the aws-sdk in the package.json. In my case, it is "aws-sdk": "2.1017.0". I am using node18, so aws-sds v2 is not included in the AWS node image by default, so I had to install it explicitly. See AWS docs. Some dependencies in package.json can also depend on aws-sdk, so you need to double-check which version you run.
• stop using aws-xray-sdk-core. In my case, I had to stop using it with the SQS client.