While using django restframework Multiple database, serializer.is_valid() always go to default database for validation

Question

I am using django restframework and want to handle multiple databases. I am using django function using(alias) and switch_db(alias) for manually switching between database whenever I want to Get, Post or update data.

I am facing problem while posting and updating data.i.e whenever serializer.is_valid() will be called.

serializer.is_valid() will go and first check for db_alias in model.py file. If I have not specified db_alias under meta it will select default database for validation. If I am specifying db_alias in model it will select that database for validation.

But I do not want to specify db_alias in model since my usecase is to store data on different database based on some logic in my view file. So dynamically I want to select database from view and want to store data in it.

Almost I have implemented but I am facing problem when my model is having Reference Field. In this case serializer.is_valid is going to default database for validating that reference field.

Required Details: I am using mongoengine(0.9.0), document, document serializer.

My files are as below:

model.py:

class ngroup(Document):

    groupname = StringField(max_length=100, required=True)
    description = StringField(max_length=100, required=False)
    parent = ReferenceField('ngroup',null=True)
    created_ts = DateTimeField(default=datetime.now())
    modified_ts = DateTimeField(default=datetime.now())
    is_deleted = BooleanField(default=False)

serializer.py:

from device_management.models import ngroup
from rest_framework_mongoengine.serializers import DocumentSerializer
from mongoengine import EmbeddedDocumentField, ReferenceField, StringField, ObjectIdField, IntField, BooleanField, FloatField, DateTimeField,ListField


class ngroupSerializer(DocumentSerializer):

    class Meta:
        model = ngroup

    def setOrgId(self, orgid):
        self.orgid = orgid

    def create(self, validated_data):
        ngroup_data = ngroup(**validated_data).switch_db(self.orgid)
        ngroup_data.save()
        return ngroup_data

    def update(self, instance, validated_data):
        ngroup_data = ngroup.objects.using(self.orgid).get(id = instance.id)
        ngroup_data = ngroup_data.switch_db(self.orgid)
        ngroup_data = ngroup_data.update(**validated_data)
        return validated_data

    def to_internal_value(self, data):
        print "data:" , data
        return super(DocumentSerializer, self).to_internal_value(data)  

view.py:

def create(self, request, format=None):
    orgid = str(request.user.orgid.id)
    data=request.data

    serializer = ngroupSerializer(data=data)
    if serializer.is_valid():
        try:
            serializer.save()
        except Exception as e:
            log.error("create" , extra={'extra':{'error': str(e),'message' :strings.DATA_VALIDATION_ERROR }})
            return response.errorResponse(message=strings.SERIALIZATION_ERROR_MSG,error=str(e),rstatus=status.HTTP_400_BAD_REQUEST)
        return response.successResponse(res_data=serializer.data, message=strings.POST_SUCCESS_MSG, rstatus=status.HTTP_201_CREATED)
    log.error("create" , extra={'extra':{'error': serializer.errors,'message' :strings.DATA_VALIDATION_ERROR }})
    return response.errorResponse(message=strings.DATA_VALIDATION_ERROR,error=serializer.errors,rstatus=status.HTTP_400_BAD_REQUEST)

settings.py:

DATABASES = {
     'default': {
        'ENGINE': 'django_mongodb_engine',
        'NAME': 'mydb',
        'USER': 'admin',
        'PASSWORD':'admin123',
        'HOST': '127.0.0.1',
        'PORT': 27017,
        'DBTYPE' : "mongo",
    },
    '586e47c784413825f2b5bc49': {
        'ENGINE': 'django_mongodb_engine',
        'NAME': 'mydb1',
        'USER': 'admin',
        'PASSWORD':'admin123',
        'HOST': '127.0.0.1',
        'PORT': 27017,
        'DBTYPE' : "mongo",
    },
    # Enter super_user organisation here. This DB will be same as default db only always
    '58996fb28441384430dc8ae6': {
        'ENGINE': 'django_mongodb_engine',
        'NAME': 'mydb',
        'USER': 'admin',
        'PASSWORD':'admin123',
        'HOST': '127.0.0.1',
        'PORT': 27017,
        'DBTYPE' : "mongo",
    },
}

pip freeze(Installation versions):

Django==1.5.11
django-browserid==2.0.2
django-classy-tags==0.8.0
django-missing==0.1.18
django-mongo-auth==0.1.3
django-mongodb-engine==0.6.0
django-mongoengine==0.2.1
django-redis-sessions==0.5.6
django-rest-framework-mongoengine==3.3.0
django-sekizai==0.10.0
django-websocket-redis==0.4.7
djangorestframework==3.1.2
djangorestframework-jwt==1.9.0
djangotoolbox==1.8.0
gevent==1.1.2
greenlet==0.4.10
httplib2==0.9.2
mongoengine==0.9.0
oauthlib==2.0.1
pika==0.10.0
Pygments==2.1.3
PyJWT==1.4.2
pymongo==2.8
python-dateutil==2.6.0
python-openid==2.2.5
pytz==2016.10
redis==2.10.5
requests==2.12.3
requests-oauthlib==0.7.0
rest-condition==1.0.3
six==1.10.0
tweepy==3.5.0
twilio==5.7.0

I have overide create in serializer to take care of database while calling serializer.save() but how to handle serializer.is_valid().

My project has been stuck at this point. Any help will be greatly appreciated...

Answer 1

Husain,

Unfortunately, you're mixing incompatible projects together. Mongoengine, django-mongoengine and Django-REST-Framework-Mongoengine projects are alternative to django-mongodb-engine, they are not meant to be used together.

As far as I know, django-mongodb-engine project's been dead for 2 years, or even longer, to be honest. At the same time, Mongoengine stack is working in production, though, development is not too active. I really want to create a proper django database backend out of Mongoengine to make it a first-class citizen in Django world, and it seems like Django guys are looking in that direction, too.

You might also want to look into this post.

This is my second attempt. I tried to switch the database connection in view create(). Didn't work for me:

settings.py

# We define 2 Mongo databases - default (project) and project2
MONGODB_DATABASES = {
    "project": {
        "name": "project",
        "host": "localhost",
        "port": 27017,
        "tz_aware": True,  # if you use timezones in django (USE_TZ = True)
    },

    "project2": {
        "name": "project2",
        "host": "localhost",
        "port": 27017,
        "tz_aware": True,  # if you use timezones in django (USE_TZ = True)
    }
}


mongoengine.register_connection(alias='default', name=MONGODB_DATABASES["project"]["name"], host="local")
mongoengine.register_connection(alias='project2', name=MONGODB_DATABASES["project2"]["name"], host="local")

connection = mongoengine.connect(db="project", alias='default')

views.py

class AuthorViewSet(MongoModelViewSet):
    lookup_field = 'id'
    serializer_class = AuthorSerializer

    def create(self, request, format=None):
        global Author
        mongoengine.connection.disconnect(alias='project')
        mongoengine.connect('project2', alias='project2')
        return super(AuthorViewSet, self).create(request, format)

    def get_queryset(self):
        return Author.objects.all()

Answer 2

This is not the exact solution to above problem but we have 2 options.

1) Do not go for serializer.is_valid() or serializer.save(). Directly create ngroup:

def my_create(self, validated_data):
    gateway = Gateway(**validated_data).switch_db(self.orgid)
    gateway.save()
    return gateway

2) Another solution is to use django-mogodb-engine and django models and modelserializers instead of documents and documents serializers.

I have tried following this with Django-mongodb-engine and are working well:

-> JWT authentication
-> custom user
-> foreign key
-> embedded model
-> list of embedded model
-> dict field
-> **Routers for switching between databases.(Manual switching DB is not required)**

I can also use middleware_classes to specify runtime in each request which database to use. Reference Link: Django Authenticate Backend Multiple Databases