What's the best way to check the data is valid before adding it to DynamoDB

Question

As I understand there are no foreign key constraints in Dynamo.

We have users, groups.

Since users and groups are tightly linked to each other, they are being stored in the same table in this particular use case. But generally speaking, it could be attributed to the same table or a different table.

Forex:

A request to add a user to a group with a given groupId.

A request to add a site to a group.

What's the best and optimized way to check that the given groupId that we will be adding to the user actually exists. The same goes for the site with a given siteId that needs to be added to a given group.

Do we send out a checkIfExists dynamoDB query for every such field before inserting the data in the DB? That sounds unoptimized and ugly. How can this be approached in a better way?

Answer 1

Do we send out a checkIfExists dynamoDB query for every such field before inserting the data in the DB? That sounds unoptimized and ugly.

This is likely the solution. I think you think it sounds "unoptimized and ugly" because you're not used to doing it, instead relying on the database to do it for you. What would you do if you needed to make sure some something that wasn't in your database exists? For example, say you use a 3rd party API for something, and you need to be sure there is a record there before creating your record? You'd have to make a request to that service first. What DynamoDB is doing is forcing you to include your business logic in your code. As for the performance, a request for a single item in DynamoDB typically takes less than 10ms, and if that's not fast enough you can use DAX and get that down to under 1ms in most cases.

Answer 2

You can use a transaction to do this, here's an example in Python:

import boto3
import botocore.exceptions

TABLE_NAME = "transaction"

def create_table():
    ddb = boto3.client("dynamodb")
    ddb.create_table(
        AttributeDefinitions=[
            {"AttributeName": "PK", "AttributeType": "S"},
            {"AttributeName": "SK", "AttributeType": "S"},
        ],
        TableName=TABLE_NAME,
        KeySchema=[{"AttributeName": "PK", "KeyType": "HASH"}, \
                  {"AttributeName": "SK", "KeyType": "RANGE"}],
        BillingMode="PAY_PER_REQUEST",
    )


def add_group(group_name: str):
    item = {
        "PK": f"G#{group_name}",
        "SK": "META",
        "groupName": group_name
    }
    boto3.resource("dynamodb").Table(TABLE_NAME).put_item(Item=item)

def add_user_to_group(user_name: str, group_name: str):

    print(f"Attempting to add user {user_name} to group {group_name}")
    
    membership_item = {
        "PK": {"S": f"G#{group_name}"},
        "SK": {"S": f"U#{user_name}"}
    }

    client = boto3.client("dynamodb")
    try:
        client.transact_write_items(
            TransactItems=[
                {
                    "ConditionCheck": {
                        "TableName": TABLE_NAME,
                        # Check if an item with this key exists
                        "Key": {
                            "PK": {"S": f"G#{group_name}"},
                            "SK": {"S": "META"}
                        },
                        "ConditionExpression": "attribute_exists(PK)"
                    }
                },
                {
                    "Put": {
                        "Item": membership_item,
                        "TableName": TABLE_NAME
                    }
                }
            ]
        )
    except botocore.exceptions.ClientError:
        return f"Group {group_name} doesn't exist!"

    return f"Added user {user_name} to group {group_name}"

if __name__ == "__main__":
    create_table()
    add_group("dummy")
    print(add_user_to_group("test", "dummy"))
    print(add_user_to_group("test", "does_not_exist"))

Basically you create a transaction with two items:

1. The ConditionCheck to test if the group exists
2. The Put/Update to create your membership

If you run the code it will create a table, add a group and then attempt to add a user to the group. That works, then it adds the user to another group that doesn't exist, which fails.

Output:

Attempting to add user test to group dummy
Added user test to group dummy
Attempting to add user test to group does_not_exist
Group does_not_exist doesn't exist!