How to make pydantic await on a async property (tortoise-orm's reverse ForeignKey)?

Question

(MRE in the bottom of the question)

In tortoise-orm, we have to await on reverse ForeignKey field as such:

comments = await Post.get(id=id).comments

But in fastapi, when returning a Post instance, pydantic is complaining:

pydantic.error_wrappers.ValidationError: 1 validation error for PPost
response -> comments
  value is not a valid list (type=type_error.list)

It makes sense as comments property returns coroutine. And I had to use this little hack to get aronud:

post = Post.get(id=id)
return {**post.__dict__, 'comments': await post.comments}

However, the real issue is when I have multiple relations: return a user with his posts with its comments. In that case I had to transform into dict my entiry model in a very ugly way (which doesn't sound good).

Here is the code to reproduce (tried to keep it as simple as possible):

models.py

from tortoise.fields import *
from tortoise.models import Model
from tortoise import Tortoise, run_async

async def init_tortoise():
    await Tortoise.init(
        db_url='sqlite://db.sqlite3',
        modules={'models': ['models']},
    )
    await Tortoise.generate_schemas()

class User(Model):
    name = CharField(80)

class Post(Model):
    title = CharField(80)
    content = TextField()
    owner = ForeignKeyField('models.User', related_name='posts')

class PostComment(Model):
    text = CharField(80)
    post = ForeignKeyField('models.Post', related_name='comments')

if __name__ == '__main__':
    run_async(init_tortoise())

__all__ = [
    'User',
    'Post',
    'PostComment',
    'init_tortoise',
]

main.py

import asyncio
from typing import List

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

from models import *


app = FastAPI()

asyncio.create_task(init_tortoise())

# pydantic models are prefixed with P
class PPostComment(BaseModel):
    text: str

class PPost(BaseModel):
    id: int
    title: str
    content: str
    comments: List[PPostComment]
    class Config:
        orm_mode = True

class PUser(BaseModel):
    id: int
    name: str
    posts: List[PPost]
    class Config:
        orm_mode = True

@app.get('/posts/{id}', response_model=PPost)
async def index(id: int):
    post = await Post.get_or_none(id=id)
    return {**post.__dict__, 'comments': await post.comments}

@app.get('/users/{id}', response_model=PUser)
async def index(id: int):
    user = await User.get_or_none(id=id)
    return {**user.__dict__, 'posts': await user.posts}

/users/1 errors out with:

pydantic.error_wrappers.ValidationError: 1 validation error for PUser
response -> posts -> 0 -> comments
  value is not a valid list (type=type_error.list)

Also you may wish to put this into init.py and run:

import asyncio
from models import *

async def main():
    await init_tortoise()
    u = await User.create(name='drdilyor')
    p = await Post.create(title='foo', content='lorem ipsum', owner=u)
    c = await PostComment.create(text='spam egg', post=p)

asyncio.run(main())

What I want is to make pydantic automatically await on those async fields (so I can just return Post instance). How is that possible with pydantic?

---

Changing /posts/{id} to return the post and its owner without comments is actually working when using this way (thanks to @papple23j):

    return await Post.get_or_none(id=id).prefetch_related('owner')

But not for reversed foreign keys. Also select_related('comments') didn't help, it is raising AttributeError: can't set attribute.

Answer 1

Sorry, I was sooo dumb.

One solution I though about is to use tortoise.contrib.pydantic package:

PPost = pydantic_model_creator(Post)
# used as
return await PPost.from_tortoise_orm(await Post.get_or_none(id=1))

But as per this question, it is needed to initialize Tortoise before declaring models, otherwise Relation's wont be included. So I was tempted to replace this line:

asyncio.create_task(init_tortoise())

...with:

asyncio.get_event_loop().run_until_complete(init_tortoise())

But it errored out event loop is already running and removing uvloop and installing nest_asyncio helped with that.

---

The solution I used

As per documentation:

Fetching foreign keys can be done with both async and sync interfaces.

Async fetch:

events = await tournament.events.all()

Sync usage requires that you call fetch_related before the time, and then you can use common functions.

await tournament.fetch_related('events')

After using .fetch_related) (or prefetch_related on a queryset), reverse foreign key would become an iterable, which can be used just as list. But pydantic would still be complaining that is not a valid list, so validators need be used:

class PPost(BaseModel):
    comments: List[PPostComment]

    @validator('comments', pre=True)
    def _iter_to_list(cls, v):
        return list(v)

(Note that validator can't be async, as far as I know)

And since I have set orm_mode, I have to be using .from_orm method :

return PPost.from_orm(await Post.get_or_none(id=42))

Remember, a few hours of trial and error can save you several minutes of looking at the README.

Answer 2

You can try using prefetch_related()

For example:

@app.get('/posts/{id}', response_model=PPost)
async def index(id: int):
    post = await Post.get_or_none(id=id).prefetch_related('comments')
    return {**post.__dict__}

Answer 3

(The following text is translated using DeepL)

There is a way to do this, but it is a bit tricky

First split the pydantic models snippet into schemas.py

from pydantic import BaseModel
from typing import List


# pydantic models are prefixed with P
class PPostComment(BaseModel):
    text: str
    class Config:
        orm_mode = True # add this line

class PPost(BaseModel):
    id: int
    title: str
    content: str
    comments: List[PPostComment]
    class Config:
        orm_mode = True

class PUser(BaseModel):
    id: int
    name: str
    posts: List[PPost]
    class Config:
        orm_mode = True

Next, rewrite models.py

from tortoise.fields import *
from tortoise.models import Model
from tortoise import Tortoise, run_async

from schemas import *

async def init_tortoise():
    await Tortoise.init(
        db_url='sqlite://db.sqlite3',
        modules={'models': ['models']},
    )
    await Tortoise.generate_schemas()

class User(Model):
    name = CharField(80)
    _posts = ReverseRelation["Post"] #1

    @property
    def posts(self): #3
        return [PPost.from_orm(post) for post in self._posts]

class Post(Model):
    title = CharField(80)
    content = TextField()
    owner = ForeignKeyField('models.User', related_name='_posts') #2
    _comments = ReverseRelation["PostComment"] #1
    
    @property
    def comments(self): #3
        return [PPostComment.from_orm(comment) for comment in self._comments]

class PostComment(Model):
    text = CharField(80)
    post = ForeignKeyField('models.Post', related_name='_comments') #2


if __name__ == '__main__':
    run_async(init_tortoise())

__all__ = [
    'User',
    'Post',
    'PostComment',
    'init_tortoise',
]

where

#1: Use ReverseRelation to declare the reverse field, here use the prefix of the bottom line to differentiate

#2: Modify the related_name

#3: Write a property function and return the corresponding pydantic model list, here you don't need to use await because the default is to access it with prefetch_related()

Finally, the main.py

import asyncio
from typing import List

from fastapi import FastAPI, HTTPException

from models import *
from schemas import *
from tortoise.query_utils import Prefetch

app = FastAPI()

asyncio.create_task(init_tortoise())

@app.get('/posts/{id}', response_model=PPost)
async def index(id: int):
    post = await Post.get_or_none(id=id).prefetch_related('_comments') #1
    return PPost.from_orm(post) #2

@app.get('/users/{id}', response_model=PUser)
async def index(id: int):
    user = await User.get_or_none(id=id).prefetch_related(
        Prefetch('_posts',queryset=Post.all().prefetch_related('_comments')) #3
    )
    return PUser.from_orm(user) #2

where

#1: Use prefetch_related() to prefetch related data

#2: For a tortoise model with orm_mode = True, you can use from_orm to convert it to a pydantic model.

#3: For multi-layer correlation data structure, you need to write another layer of prefetch_related()