aws java: parse S3Put event coming from SQS to lambda

Question

It's been a surprise for me to see that handle event sources is something handcrafty...

I mean, I've been strugling for some days to try to receive ("deserialize") an S3Put event into my handler (using spring-cloud-function-aws-adapter Consumer).

Currently, I'm using an S3(send put event) -> SQS(send SQSMessage) -> lambda(receive SQSMessage).

My current code Lambda function (handler) is:

@Component
@RequiredArgsConstructor
@Slf4j
public class PushedDocumentLambdaConsumer implements Function<SQSEvent, String> {

    @Override
    public String apply(SQSEvent sqsEvent) {
        log.info("Found " + sqsEvent.getRecords().size() + " sqs event(s)");
        for (SQSEvent.SQSMessage sqsMessage : sqsEvent.getRecords()) {
            log.info("sqs message body: {}", sqsMessage.getBody());

            final String body = sqsMessage.getBody().replace("Records", "records");
            log.info("Replaced sqs message body: {}", body);

            Gson gson = new Gson();
            S3Event s3Event = gson.fromJson(sqsMessage.getBody(), S3Event.class);
            log.info("s3Event.toString(): {}", s3Event.toString());

            if (s3Event != null && s3Event.getRecords() != null) {
                log.info("Number of records: {}", s3Event.getRecords().size());
                for (S3EventNotificationRecord s3EventNotificationRecord : s3Event.getRecords()) {
                    log.info("event name: {}", s3EventNotificationRecord.getEventName());
                    log.info("event source: {}", s3EventNotificationRecord.getEventSource());

                    S3Entity s3Entity = s3EventNotificationRecord.getS3();
                    if (null != s3Entity) {
                        log.info("s3Entity bucket name: {}", s3Entity.getBucket().getName());
                        log.info("s3Entity object name: {}", s3Entity.getObject().getKey());
                    }
                }
            }
        }

        return "Ok";
    }

}

What I'm currently receiving is:

{
   "Records":[
      {
         "body":"{\"Records\": [{\"eventVersion\": \"2.1\", \"eventSource\": \"aws:s3\", \"awsRegion\": \"us-east-1\", \"eventTime\": \"2024-03-01T12:54:12.608Z\", \"eventName\": \"ObjectCreated:Put\", \"userIdentity\": {\"principalId\": \"AIDAJDPLRKLG7UEXAMPLE\"}, \"requestParameters\": {\"sourceIPAddress\": \"127.0.0.1\"}, \"responseElements\": {\"x-amz-request-id\": \"b0c98e62\", \"x-amz-id-2\": \"eftixk72aD6Ap51TnqcoF8eFidJG9Z/2\"}, \"s3\": {\"s3SchemaVersion\": \"1.0\", \"configurationId\": \"abdde1c9\", \"bucket\": {\"name\": \"espaidoc\", \"ownerIdentity\": {\"principalId\": \"A3NL1KOZZKExample\"}, \"arn\": \"arn:aws:s3:::espaidoc\"}, \"object\": {\"key\": \"580d1374-2ec1-4bb3-8c14-2c43d07c7833\", \"sequencer\": \"0055AED6DCD90281E5\", \"size\": 2333, \"eTag\": \"1d11469e9d81f07729548d7708bbab82\"}}}]}",
         "receiptHandle":"ZGJhNTJjNDItODU1Yy00ZjY2LWI4MmUtY2RmNDRhNzg5YTRkIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6U3FzUXVldWVDcmVhdGUgMmI1YjY3OTctMjY0Yy00NTVlLTgyNzMtZDIwMDlkYWVhZDliIDE3MDkyOTc2NTMuNDg2MTU4",
         "md5OfBody":"2bb9d9a1a018099143bd02c0aea8b0cd",
         "eventSourceARN":"arn:aws:sqs:us-east-1:000000000000:SqsQueueCreate",
         "eventSource":"aws:sqs",
         "awsRegion":"us-east-1",
         "messageId":"2b5b6797-264c-455e-8273-d2009daead9b",
         "attributes":{
            "SenderId":"000000000000",
            "SentTimestamp":"1709297652665",
            "ApproximateReceiveCount":"1",
            "ApproximateFirstReceiveTimestamp":"1709297653486"
         },
         "messageAttributes":{
            
         }
      }
   ]
}

As you can see, I'm trying to deserialize body to an S3Event using Gson, but I've realized that it gets me an S3Event without Records...

1. Is not there any library engaging for this parsing issues?
2. How could I parse it "manually"?

Answer 1

I believe the problem is you're trying to use gson to parse the SQS message into an S3Event, when it is actually an S3EventNotification. It looks like you might not even have to parse it with gson, you could do this:

S3EventNotification s3EventNotification = S3EventNotification.parse(sqsMessage.getBody());
for (S3EventNotification.S3EventNotificationRecord record : s3EventNotification.getRecords())
{
    S3EventNotification.S3Entity s3Entity = record.getS3();
    if (null != s3Entity)
    {
        log.info("s3Entity bucket name: {}", s3Entity.getBucket().getName());
        log.info("s3Entity object name: {}", s3Entity.getObject().getKey());
    }
}