Reading different types of events from SNS + SQS

Question

I am working on creating a system where I need to read several different types of events from an SNS + SQS combination i.e. there is an existing system which puts different types of events into SNS topic and there is an SQS subscribed to this SNS topic. After that there is a Lambda (L1) getting triggered by SQS.

Below are two sample SQS events out of many:

{
    "event_type" : "started",
    "parent_id" : "a1",
    "timestamp" : "t1"
}

{
    "event_type" : "completed",
    "parent_id" : "a1",
    "child_id" : "a2",
    "timestamp" : "t2"
}

On getting these events L1 keeps storing these events as below in a NoSQL store (e.g. DynamoDB) (grouped by parent_id and ordered by timestamp i.e. a1 --> List):

{
    "a1" : [
                {
                    "event_type" : "started",
                    "parent_id" : "a1",
                    "timestamp" : "t1"
                },
                {
                    "event_type" : "completed",
                    "parent_id" : "a1",
                    "child_id" : "a2",
                    "timestamp" : "t2"
                }
            ]
}

Q: Could you please help me with the low level data modelling I should be doing inside L1 for these events. Should I even try to model it myself or just use Object Mapper (Gson or Jackson) to do the job?

Further there is another Lambda (L2) which reads the events corresponding to the given parent_id (from DB), makes a service call to fetch additional metadata for all the events, prepares another event list with these additional fields and returns to the caller.

e.g.

        {
            "event_type" : "started",
            "parent_id" : "a1",
            "timestamp" : "t1",
            "metadata" : {
                "m1" : "abc",
                "m2" : "xyz"
            }
        },
        {
            "event_type" : "completed",
            "parent_id" : "a1",
            "child_id" : "a2",
            "timestamp" : "t2",
            "metadata" : {
                "m3" : "lmn"
            }
        }

Could you please give suggestions towards designing this system.

Any suggestions would be much appreciated.

Thanks

Answer 1

For the first question, what I suggest is, you don't need to map to any Pojo, just to a map with key/value. and then based on the event_type, you can perform your own business logic such as store to DynamoDB.

And then you can enable Streaming with DynamoDB for another lambda (Lambda2) to continue with next steps.