How can I implement a transaction of 50 writes in dynamoDB?

Question

I’m aware there is a hard limit of 25 items per transaction. However, I’m sure there is a way of implementing transactions for more items from scratch. How might I go about it?

I’m thinking something like, keep a version number on every item. Fetch all the items up front, during insert verify version number is the same. Ie optimistic locking. If the condition fails, revert all failed items. Naturally, I can imagine that the revert could fail and I need to do optimistic locking on the revert and end up in a deadlock of reverts.

Answer 1

The solution I found in the end was to implement pessimistic locking. It supports an arbitrary number of writes and reads and guarantees transactional consistency. The catch is if you're not careful, it's easy to run into deadlocks.

The idea is that you

1. Create a lock table. Each row refers to a specific lock. The primary key of the lock table should be a string which I'll refer to as the lock-key. Often you'll want to lock a specific entity so this is a reasonable format for the lock-key {table_name}#{primary_key} but it might be more arbitrary so any string will do. Rows in the lock table should also auto-delete after a certain time period as per a ttl field ie TimeToLiveSpecification.
2. Before starting the transaction, acquire the lock. You do this by creating the row with your arbitrary lock-key and with a conditional check that the row doesn't already exist. If it does exist the row creation should fail which means another process has already acquired the lock. You then need to poll, trying to recreate the lock row until the lock has been released.
3. Once you have acquired the lock you need to keep the lock alive with a heartbeat to prevent other tasks from executing. The hearbeat process should increment a heartbeat property on the lock row which reflects the last-active time of the lock. The ttl of the row should be greater than the heartbeat interval. Normally about double, so that the lock is not auto-purged erroneously. If your process dies, the lock will be naturally released by the auto-deletion of the ttl.
4. If your task completes successfully it should delete the lock row freeing it up for other tasks.