Dynamodb to starrocks etl

Question

I need to etl a set of tables from DynamoDB to StarRocks. Has anyone used the StarRocks Load tool to accomplish this? If so, can you share how?

Second related question: some of the tables are very large (500,000,000 records). This is live data. I expect the etl to StarRocks to take more than 24 hours. If it does, how does one both etl the data present at start amd then keep track of deltas to the data (new records are not a problem, but deletes and updates in a stream may happen in the stream before the record is put into the table.) So, how does one handle this?

Thanks in advance!

Answer 1

I don't know anything about Starrocks, never heard of it.

But for your question regarding keeping them in sync, DynamoDB supports two types of streams

• DynamoDB Streams - 24 hour retention
• Kinesis Data Streams - up to 1 year retention

So you have both options to help you replay the events of the source region while migration is happening.