django-rest-framework "This field is required" on POST

Question

Whenever I POST to my django-rest-framework (DRF) endpoints, I keep receiving a "HTTP 400 Bad Request" {"offeror_organization":["This field is required."]} response. But, given the curl example below, I'm clearly specifying a value.

This happens regardless of the Content-Type (application/json, application/x-www-form-urlencoded, multipart/form-data). The only time it works is when I submit using the "HTML form" (vs. the "Raw Data") tab on the DRF web interface.

There's a few similar SO posts (like this and this), but none of the solutions seem to be working for me.

Model:

class OrganizationManager(models.Manager):
    def get_by_natural_key(self, offeror_organization):
        return self.get(offeror_organization=offeror_organization)

class Organization(models.Model):
    idorganization = models.AutoField(primary_key=True)
    offeror_organization = models.CharField(max_length=250, null=False, blank=False, verbose_name='Offeror Organization')
    created_at = models.DateTimeField(auto_now_add=True, null=False)
    updated_at = models.DateTimeField(auto_now=True, null=False)

    objects = OrganizationManager()

    def natural_key(self):
        return "%s" % (self.offeror_organization)

    def __str__(self):
        return self.offeror_organization

Serializer:

class OrganizationSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Organization
        fields = ['offeror_organization']

    # I've tried both with and without a create function
    def create(self, validated_data): 
        organization_data = validated_data.pop('offeror_organization', None)
        if organization_data:
            organization = Organization.objects.get_or_create(**organization_data)[0]
            validated_data['offeror_organization'] = organization

views/api.py:

from webapp.models import Organization
from webapp.serializers import OrganizationSerializer

from rest_framework import viewsets

class OrganizationViewSet(viewsets.ModelViewSet):
    queryset = Organization.objects.all().order_by('offeror_organization')
    serializer_class = OrganizationSerializer

urls.py:

from django.urls import include, path
from rest_framework import routers
from . import views

router = routers.DefaultRouter()
router.register(r'organization', views.OrganizationViewSet)

urlpatterns = [
    ...
    path('api/', include(router.urls)),
    path('api-auth/', include('rest_framework.urls', namespace='rest_framework')),
]

Curl Command:

curl -X POST -H 'Content-Type: application/json' -d '{"offeror_organization":"Test2"}' 10.101.10.228:29000/webapp/api/organization/

settings.py MIDDLEWARE:

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.RemoteUserMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'csp.middleware.CSPMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware'
]

settings.py REST_FRAMEWORK

# currently have all API authentication disabled while troubleshooting this issue
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [],
    'DEFAULT_PERMISSION_CLASSES': [],
}

Answer 1

In my case, fixing this issue required "maneuvering" around a few different implementation constraints.

nginx + uWSGI Socket + Django REMOTE_USER Authentication: As mentioned in this post's comments/chat, I've got both an nginx proxy and a uWSGI application server fronting my Django application. Since I'm relying upon REMOTE_USER Authentication, my uwsgi/nginx configuration must use uWSGI sockets (vs. http) so that I may pass the REMOTE_USER from nginx to Django as an environment variable. When using http (coupled w/ nginx proxy_pass), although proxy_pass can set headers or cookies, those seemingly cannot translate over to Django (which requires the environment variable).

I think there's a few issues at play when trying to POST to a Django/DRF application served using uWSGI sockets. Per the uWSGI Things to know (best practices), "TL/DR: if you plan to expose uWSGI directly to the public, use --http, if you want to proxy it behind a webserver speaking http with backends, use --http-socket". In my case, having both a web application and a DRF-based API (that I want other services and systems to talk to), I need both! As a (hopefully temporary) workaround, I'm currently spawning two uWSGI processes - one using --socket, and one using --http (for API POST calls). If you POST while using ---socket, you'll likely get an Empty Response error from DRF.

As an aside, I initially saw some "promise" in utilizing uwsgi_curl (from uwsgi_tools) to POST over the uWSGI socket (which resulted in the "field is required" error (vs. the Empty Response error), but that's when I started to run into my second issue...

POST nested application/json w/ simultaneous file upload: The "Organization" model referenced in the post was mostly proof-of-concept, as it's the least complex model in my Django application. In reality, I need to post to a more complex model with nested serialization, as the model contains Foreign Key's to other models. But that's totally do-able with DRF. Except in my case, where one of my model attributes is a FileUpload field. As noted in other SO Questions (like this one), there's also a few issues in trying to POST nested (i.e. not "flat") application/json with a file upload in a single request. While I was never able to fully understand the issue at play (at least using drf_writable_nested.serializers.WritableNestedModelSerializer in my case), I simplified the problem at-hand by writing my own custom Serializer (serializers.Serializer), this way I could avoid nested JSON objects (like { "offeror_organization": {"offeror_organization: "Test"}} in my POST requests. This fixed my issue.

With the custom serializer in place to mitigate the nested JSON + file upload issue, I bet the uwsgi_curl POST would work. Although then external client systems/services are limited to using that Python package. Anyways, I'll update my answer once I try it out. Thanks to @Michael for his comments and helping to lead me down the right "road".

Answer 2

I have the same setup (nginx + gunicorn + django + rest-framework + drf-writeable-nested) but I could figure out a valid format for the POST request containing multipart/form-data:

It needs to be like this:

json:

{
   'name': 'test-name',
   'files': [
      {
         'file': 'test-file-1'
      },
      {
         'file': 'test-file-2'
      },
      ...
   ],
   ...
}

must be formatted to:

FormData:

name: test-name
files[0]file: test-file-1
files[1]file: test-file-2
...

Some libraries would use a dot after the brackets for nested lists, which would lead to the This field is required error. Or even another bracket after the list bracket would lead to the same error. This is wrong: files[0].file This is also wrong: files[0][file]

My example assumes the following Django-classes:

# views.py

from rest_framework.viewsets import ModelViewSet
from rest_framework.parsers import MultiPartParser, FormParser
from .serializers import YourCustomModelSerializer

class YourCustomModelViewSet(ModelViewSet):
    queryset = YourCustomModel.objects.all()
    parser_classes = [FormParser, MultiPartParser]
    permission_classes = []
    serializer_class = YourCustomModelSerializer

# serializers.py

from rest_framework.serializers import ModelSerializer
from drf_writable_nested.serializers import WritableNestedModelSerializer
from .models import YourCustomModel, File

class FileSerializer(ModelSerializer):
    class Meta:
        model = File
        fields = ['file']

class YourCustomModelSerializer(WritableNestedModelSerializer):
    # Reverse foreign key
    files = FileSerializer(many=True)

    class Meta:
        model = YourCustomModel
        read_only_fields = ('id', )
        fields = [
            'id',
            'name',
            'files'
        ]

# models.py

from django.db import models

class File(models.Model):
    file = models.FileField()

class YourCustomModel(models.Model):
    name = models.CharField(max_length=200)

I used the following javascript/typescript frontend code to pack my json data into a FormData request:

const requestBody = {
  name: 'test-name',
  files: [
    { file: file1 }, // File object
    { file: file2 }, // File object
    { file: file3 }  // File object
  ]
}

// # use your own code to serialize the above javascript dict/json/object
// into form-data object (I used the library https://www.npmjs.com/package/object-to-formdata but I had to write a patch for the exact reason described above: files[0].file is not correctly parsed by Django and files[0][file] also won't work, therefore I slightly changed the library code so that it will format my object to files[0]file for every nested item:

// 2. prepare a serializer-library to convert the dict into a special form which can be parsed by django.
const options = {
  indices: true,
  allowEmptyArrays: true,
  useDotOrBracketsOrNothingForNestedObjects: 2 // Option 0: DOT-Notation, 1: Brackets, 2: Nothing (this option is from my custom patch)
}

// use npx patch: https://stackoverflow.com/a/62567504/3433137
// (I patched this serialize library and the patch is somewhere stored as a file in this project)
const formData = serialize(
  requestBody,
  options
)
    
// 3. upload the data
api.post(url, formData)