Memory leak in Ajax implementation of a file uploader (Django on Heroku)

Question

I've created an ajax file uploader for Django, but each file uploaded takes up a big piece of memory (30-80 mb), and doesn't seem to let it go.

We're on Heroku, which allots 512mb of memory per dyno, so I quickly start getting over-memory errors.

Here is the Django view code to handle the request:

if request.is_ajax():
        # the file is stored raw in the request
        upload = request
        is_raw = True
        try:
          filename = request.GET[ 'add_image' ]
        except KeyError:
          return HttpResponseBadRequest( "AJAX request not valid" )
        (fileBaseName, fileExtension)=os.path.splitext(filename)

        uniquename = biz_id + "__" + get_a_uuid() + fileExtension
        saved = save_upload(upload, uniquename, biz)

And here is the save_upload code:

try:
    #BusinessImage is my Django model.  It uses django-imagekit to processs
    #the raw uploaded image into three sizes (plus the original)
    bi = BusinessImage(name=uploaded.GET.get("name"), business=biz)
    if not BusinessImage.objects.filter(business=biz).exists():
        bi.primary_image = True
    bi.original_image.save(filename,ContentFile(uploaded.read()))
except IOError:
    # could not open the file most likely
    return False
finally:
    uploaded.close()
return True

This code is adapted from this post (thanks to Alex Kuhl and Thunder Rabbit).

I'm thinking the memory problem might have something to do with django-imagekit or I may not be closing the files properly, but I'm not sure. I'd really appreciate any help.

Thanks!

Clay

Answer 1

Django upload handlers are not generally for processing and trasforming files, they are for getting them ready to pass to views in the request.FILES array (via saving to /tmp or saving to memory).

Try using the [nginx upload module][1] (and upload progress) if you want an easy, quick, and low memory way to get progress feedback. It copies the upload to a specified location on disk and passes the request onto the view with the file path, size, and mime type in POST vars. Much more efficient than doing so with django.