Index DynamoDB streams to elastic search

Question

I have a requirement for implementing following entities in a DynamoDB table

I have stored these entities in DynamoDB as below.

• Partition Key : PROJ#ProjectId:CountryId
• Sort Key : Project Name
• Company : company data as JSON document

Since this is a one to many relationship, N number of projects of the same company will create N number of project records and same company details will be stored in their Company attribute. The reason for doing this is, the most critical data access point is via ProjectId and CountryId (Assume that I can't change this DB design)

I have a requirement to implement a search functionality which supports filter table using company name, address, project name, country etc (using a single filter or any combination of these filters). I'm using DynamoDB streams to feed elastic search cluster and update any creation, deleting or update of the details there and use elastic search API to query data.

But I need to index these data in following format, so that when I receive the details from elastic search, data will not be duplicated

{
    "id" : 1
    "name" : "ABC",
    "description" : "description",
    "address" : "address",
    "projects" : [
        {
            "id" : 10,
            "name" : "project 1",
            "countryId" : 10
        },
        {
            "id" : 20,
            "name" : "project 1",
            "countryId" : 10
        }
    ]
}

At the record creation time, since Project records are creating as single records, is there any recommended or standard way that I can grab all the Project records of Company and create the above json document and index it in elastic search?

Answer 1

This is how I would approach it :

1. In elastic the document id will be the companyID
2. What you can do is create a lambda that is triggered based on the change streams and use elastic's update by query to query for the document and PAINLESS scripting to update the project section of the document, this will work for less frequent changes.