Dictionary serialization for DynamoDB ExclusiveStartKey not working

Question

For a paginated response from Dynamo, I am attempting to preserve the ExclusiveStartKey value. In the code sample, if I use the response.LastEvaluatedKey value directly, subsequent requests work fine.

However, if I serialize the response.LastEvaluatedKey and then serialize it back to be used as the ExclusiveStartKey value, subsequent requests fail with the following error message:

The provided starting key is invalid: One or more parameter values were invalid: Null attribute value types must have the value of true

The deserialized dictionary appears to have the same values as the original dictionary...is there anything to check to see what is different between the two?

QueryResponse response = null;

do
{

    string gsiPartitionKey = "gsi-pk-value-1";

    var queryRequest = new QueryRequest()
    {
        TableName = "my-table",
        IndexName = "my-index",
        KeyConditionExpression = "IndexPk = :s_gsiPartitionKey",
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
        {
            {
                ":s_gsiPartitionKey", new AttributeValue { S = gsiPartitionKey}
            }
        },
        Limit = 1
    };

    if (response != null)
    {
        //OPTION 1 - OK - Using LastEvaluatedKey directly works fine
        //queryRequest.ExclusiveStartKey = response.LastEvaluatedKey;

        //OPTION 2 - BAD - Serializing and deserializing  fails
        var serialized = JsonConvert.SerializeObject(response.LastEvaluatedKey);
        var deserialized = JsonConvert.DeserializeObject<Dictionary<string, AttributeValue>>(serialized);
        queryRequest.ExclusiveStartKey = deserialized;
    }

    response = await DynamoDbClient.QueryAsync(queryRequest);

} while (response.LastEvaluatedKey.Count != 0);

Answer 1

I hit this issue today, and thought I'd update this. Inside AttributeValue class, there is a non public member _null of type bool? that is being initialised incorrectly when de-serialising from JSON. It's being set as false when it should be set as null.

Using reflection, after deserialization, I set the value to null for each key in the dictionary and AWS now returns data as expected.

To access the private member I used this function:

public void SetPrivatePropertyValue<T>(object obj, string propName, T val)
{
    Type t = obj.GetType();

    // add a check here that the object obj and propertyName string are not null
    foreach (FieldInfo fi in obj.GetType().GetFields(BindingFlags.Instance | BindingFlags.NonPublic))
    {
        if (fi.Name.ToLower().Contains(propName.ToLower()))
        {
            fi.SetValue(obj, val);
            break;
        }
    }
}

And the method call was SetPrivatePropertyValue<bool?>(attribute, "_null", null);

Good luck!

Answer 2

Use a custom json converter.

https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-converters-how-to

In my case, I know that I'm only serializing S string instances of AttributeValue, so the converter is simple, but this could be adjusted to handle all of the available types if you needed.

The converter:

public class AttributeValueJsonConverter : JsonConverter<AttributeValue>
{
    public override AttributeValue Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options) => new AttributeValue(reader.GetString());

    public override void Write(
        Utf8JsonWriter writer,
        AttributeValue attributeValue,
        JsonSerializerOptions options) => writer.WriteStringValue(attributeValue.S);
}

You can see how I'm using this converter when I serialize the LastEvaluatedKey, which is Dictionary<string, AttributeValue> -

var jsonSerializerOptions = new JsonSerializerOptions
{
    Converters =
    {
        new AttributeValueJsonConverter()
    }
};
var serialized = JsonSerializer.Serialize(queryResponse.LastEvaluatedKey, jsonSerializerOptions);
var deserialized = JsonSerializer.Deserialize<Dictionary<string, AttributeValue>>(serialized, jsonSerializerOptions);