DynamoDB Local to DynamoDB AWS

Question

I've built an application using DynamoDB Local and now I'm at the point where I want to setup on AWS. I've gone through numerous tools but have had no success finding a way to take my local DB and setup the schema and migrate data into AWS.

For example, I can get the data into a CSV format but AWS has no way to recognize that. It seems that I'm forced to create a Data Pipeline... Does anyone have a better way to do this?

Thanks in advance

Answer 1

As was mentioned earlier, DynamoDB local is there for testing purposes. However, you can still migrate your data if you need to. One approach would be to save data into some format, like json or csv and store it into S3, and then use something like lambdas or your own server to read from S3 and save into your new DynamoDB. As for setting up schema, You can use the same code you used to create your local table to create remote table via AWS SDK.

Answer 2

you can create a standalone application to get the list of tables from the local dynamoDB and create them in your AWS account after that you can get all the data for each table and save them.

I'm not sure which language you familiar with but will explain some API might help you in Java.

DynamoDB.listTables();
DynamoDB.createTable(CreateTableRequest);

example about how to create table using the above API

ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(1L, 1L);
    try{
            CreateTableRequest groupTableRequest = mapper.generateCreateTableRequest(Group.class); //1
            groupTableRequest.setProvisionedThroughput(provisionedThroughput); //2
//        groupTableRequest.getGlobalSecondaryIndexes().forEach(index -> index.setProvisionedThroughput(provisionedThroughput)); //3
            Table groupTable = client.createTable(groupTableRequest); //4
            groupTable.waitForActive();//5
        }catch(ResourceInUseException e){
            log.debug("Group table already exist");
        }

1- you will create TableRequest against mapping 2- setting the provision throughput and this will vary depend on your requirements 3- if the table has global secondary index you can use this line (Optional) 4- the actual table will be created here 5- the thread will be stopped till the table become active

I didn't mention the API related to data access (insert ... etc), I supposed that you're familiar with since you already use them in local dynamodb

Answer 3

I did a little work setting up my local dev environment. I use SAM to create the dynamodb tables in AWS. I didn't want to do the work twice so I ended up copying the schema from AWS to my local instance. The same approach can work the other way around.

aws dynamodb describe-table --table-name chess_lobby \
| jq '.Table' \
| jq 'del(.TableArn)' \
| jq 'del(.TableSizeBytes)' \
| jq 'del(.TableStatus)' \
| jq 'del(.TableId)' \
| jq 'del(.ItemCount)' \
| jq 'del(.CreationDateTime)' \
| jq 'del(.GlobalSecondaryIndexes[].IndexSizeBytes)' \
| jq 'del(.ProvisionedThroughput.NumberOfDecreasesToday)' \
| jq 'del(.GlobalSecondaryIndexes[].IndexStatus)' \
| jq 'del(.GlobalSecondaryIndexes[].IndexArn)' \
| jq 'del(.GlobalSecondaryIndexes[].ItemCount)' \
| jq 'del(.GlobalSecondaryIndexes[].ProvisionedThroughput.NumberOfDecreasesToday)' > chess_lobby.json


aws dynamodb create-table \
--cli-input-json file://chess_lobby.json \
--endpoint-url http://localhost:8000

The top command uses describe table aws cli capabilities to get the schema json. Then I use jq to delete all unneeded keys, since create-table is strict with its parameter validation. Then I can use create-table to create the table in the local environent by using the --endpoint-url command.
You can use the --endpoint-url parameter on the top command instead to fetch your local schema and then use the create-table without the --endpoint-url parameter to create it directly in AWS.