Multiple DynamoDB(PynamoDB) Queries with FastAPI endpoint raising BrokenResourceError

Question

I'm trying to make multiple queries within one API request. I'm finding that I can make 1 query with no issues but when I try and do multiple within the same request I end up with a BrokenResourceError. I do get the return value to be what I want but because of this error it looks like my backend is hanging up which causes other Requests to wait until this request returns.

I can't find any information on this specific issue but after looking into the error. I read the following reasons for BrokenResourceError being raised.

• The resource was closed or otherwise disposed of by another task
• The resource was closed or otherwise disposed of by an external process or system
• The resource encountered an error or exception while being used, and is no longer functionalCode

I believe that the one i'm being effected by is The resource was closed or otherwise disposed of by another task.

• The resource is being shared between multiple tasks, and one of the tasks has finished using it and has closed or disposed of it.
• The resource is being used by a task that has been cancelled or interrupted, and the resource is being closed or disposed of as part of the cancellation or interruption process.
• The resource is being used by a task that is part of a group of tasks, and the group is being cancelled or interrupted, causing all of the tasks in the group to be cancelled or interrupted and their resources to be closed or disposed of.

I've tried using Boto3 alone instead of PynamoDB with no luck.

Example code below that throws the exception.

@my_router.get("/my_endpoint")
async def my_endpoint(
):
    attributes = []
    for x in range(0, 10):
        attributes.append(PynamoDBClass.get(
            hash_key="hash_key",
            range_key="range_key",
            consistent_read=True,
        ).attribute_values)
    return attributes

Used boto3 instead of pynamoDB Split the queries into multiple requests (works but not ideal for my situation) Looked into the exception and causes for the exception. Tried getting all the data in one query and i end up with the same error. :thinking: Perhaps it's that the request is taking too long.

Answer 1

I'm trying to make multiple queries within one API request.

Its not clear but it seems you have a list of partition and sort keys for which you want to retrieve the corresponding items for each of them. For this you need to use BatchGetItem:

item_keys = [('forum-{0}'.format(x), 'subject-{0}'.format(x)) for x in range(1000)]
for item in Thread.batch_get(item_keys):
    print(item)

https://pynamodb.readthedocs.io/en/stable/batch.html#batch-gets

Answer 2

I created asyncio tasks for the queries and awaited the responses

task = asyncio.create_task(get_data())
value = await task