Custom command to upload photo to Photologue from within Django shell?

Question

I have successfully employed Photologue to present galleries of regularly-created data plot images. Of course, now that the capability has been established, an obscene number of data plots are being created and they need to be shared!

Scripting the process of uploading images and adding them to galleries using manage.py from the Django shell is the next step; however, as an amateur with Django, I am having some difficulties.

Here is the custom command addphoto.py that I have currently developed:

from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
from photologue.models import Photo, Gallery

import os
from datetime import datetime
import pytz

class Command(BaseCommand):

    help = 'Adds a photo to Photologue.'

    def add_arguments(self, parser):
        parser.add_argument('imagefile', type=str)
        parser.add_argument('--title', type=str)
        parser.add_argument('--date_added', type=str, help="datetime string in 'YYYY-mm-dd HH:MM:SS' format [UTC]")
        parser.add_argument('--gallery', type=str)

    def handle(self, *args, **options):

        imagefile = options['imagefile']

        if options['title']:
            title = options['title']
        else:
            base = os.path.basename(imagefile)
            title = os.path.splitext(base)[0]
        if options['date_added']:
            date_added = datetime.strptime(options['date_added'],'%Y-%m-%d %H:%M:%S').replace(tzinfo=pytz.UTC)
        else:
            date_added = timezone.now()

        p = Photo(image=imagefile, title=title, date_added=date_added)
        p.save()

Unfortunately, when executed with --traceback, it results in the following:

./manage.py addphoto '../dataplots/monitoring/test.png' --traceback
Failed to read EXIF DateTimeOriginal
Traceback (most recent call last):
  File "/home/user/mysite/photologue/models.py", line 494, in save
    exif_date = self.EXIF(self.image.file).get('EXIF DateTimeOriginal', None)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/db/models/fields/files.py", line 51, in _get_file
    self._file = self.storage.open(self.name, 'rb')
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 300, in _open
    return File(open(self.path(name), mode))
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 405, in path
    return safe_join(self.location, name)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/utils/_os.py", line 78, in safe_join
    'component ({})'.format(final_path, base_path))
django.core.exceptions.SuspiciousFileOperation: The joined path (/home/user/mysite/dataplots/monitoring/test.png) is located outside of the base path component (/home/user/mysite/media)
Traceback (most recent call last):
  File "./manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/user/mysite/photologue/management/commands/addphoto.py", line 36, in handle
    p.save()
  File "/home/user/mysite/photologue/models.py", line 553, in save
    super(Photo, self).save(*args, **kwargs)
  File "/home/user/mysite/photologue/models.py", line 504, in save
    self.pre_cache()
  File "/home/user/mysite/photologue/models.py", line 472, in pre_cache
    self.create_size(photosize)
  File "/home/user/mysite/photologue/models.py", line 411, in create_size
    if self.size_exists(photosize):
  File "/home/user/mysite/photologue/models.py", line 364, in size_exists
    if self.image.storage.exists(func()):
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 392, in exists
    return os.path.exists(self.path(name))
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 405, in path
    return safe_join(self.location, name)
  File "/home/user/mysite/venv/lib/python3.5/site-packages/django/utils/_os.py", line 78, in safe_join
    'component ({})'.format(final_path, base_path))
django.core.exceptions.SuspiciousFileOperation: The joined path (/home/user/mysite/dataplots/monitoring/cache/test_thumbnail.png) is located outside of the base path component (/home/user/mysite/media)

Obviously, a copy of the image file was not put in the the media/ directory. Also, while the image, title, and date_added columns are populated in the photologue_photos table of the website database, the slug column is not.

How can the file be uploaded to the MEDIA_ROOT directory?

---

Here are the relevant snippets from the Photo and ImageModel models from the Photologue models.py file, for reference:

class Photo(ImageModel):
    title = models.CharField(_('title'),
                         max_length=250,
                         unique=True)
    slug = models.SlugField(_('slug'),
                        unique=True,
                        max_length=250,
                        help_text=_('A "slug" is a unique URL-friendly title for an object.'))
    caption = models.TextField(_('caption'),
                               blank=True)
    date_added = models.DateTimeField(_('date added'),
                                      default=now)
    is_public = models.BooleanField(_('is public'),
                                    default=True,
                                    help_text=_('Public photographs will be displayed in the default views.'))
    sites = models.ManyToManyField(Site, verbose_name=_(u'sites'),
                                   blank=True)

    objects = PhotoQuerySet.as_manager()

    def save(self, *args, **kwargs):
        if self.slug is None:
            self.slug = slugify(self.title)
        super(Photo, self).save(*args, **kwargs)


class ImageModel(models.Model):
    image = models.ImageField(_('image'),
                          max_length=IMAGE_FIELD_MAX_LENGTH,
                          upload_to=get_storage_path)
    date_taken = models.DateTimeField(_('date taken'),
                                  null=True,
                                  blank=True,
                                  help_text=_('Date image was taken; is obtained from the image EXIF data.'))
    view_count = models.PositiveIntegerField(_('view count'),
                                         default=0,
                                         editable=False)
    crop_from = models.CharField(_('crop from'),
                             blank=True,
                             max_length=10,
                             default='center',
                             choices=CROP_ANCHOR_CHOICES)
    effect = models.ForeignKey('photologue.PhotoEffect',
                           null=True,
                           blank=True,
                           related_name="%(class)s_related",
                           verbose_name=_('effect'))

    class Meta:
        abstract = True

    def __init__(self, *args, **kwargs):
        super(ImageModel, self).__init__(*args, **kwargs)
        self._old_image = self.image

    def save(self, *args, **kwargs):
        image_has_changed = False
        if self._get_pk_val() and (self._old_image != self.image):
            image_has_changed = True
            # If we have changed the image, we need to clear from the cache all instances of the old
            # image; clear_cache() works on the current (new) image, and in turn calls several other methods.
            # Changing them all to act on the old image was a lot of changes, so instead we temporarily swap old
            # and new images.
            new_image = self.image
            self.image = self._old_image
            self.clear_cache()
            self.image = new_image  # Back to the new image.
            self._old_image.storage.delete(self._old_image.name)  # Delete (old) base image.
        if self.date_taken is None or image_has_changed:
            # Attempt to get the date the photo was taken from the EXIF data.
            try:
                exif_date = self.EXIF(self.image.file).get('EXIF DateTimeOriginal', None)
                if exif_date is not None:
                    d, t = exif_date.values.split()
                    year, month, day = d.split(':')
                    hour, minute, second = t.split(':')
                    self.date_taken = datetime(int(year), int(month), int(day),
                                           int(hour), int(minute), int(second))
            except:
                logger.error('Failed to read EXIF DateTimeOriginal', exc_info=True)
        super(ImageModel, self).save(*args, **kwargs)
        self.pre_cache()

---

Here is the get_storage_path function, as requested:

# Look for user function to define file paths
PHOTOLOGUE_PATH = getattr(settings, 'PHOTOLOGUE_PATH', None)
if PHOTOLOGUE_PATH is not None:
    if callable(PHOTOLOGUE_PATH):
        get_storage_path = PHOTOLOGUE_PATH
    else:
        parts = PHOTOLOGUE_PATH.split('.')
        module_name = '.'.join(parts[:-1])
        module = import_module(module_name)
        get_storage_path = getattr(module, parts[-1])
else:
    def get_storage_path(instance, filename):
        fn = unicodedata.normalize('NFKD', force_text(filename)).encode('ascii', 'ignore').decode('ascii')
        return os.path.join(PHOTOLOGUE_DIR, 'photos', fn)

Answer 1

Here is the working version of the custom addphoto.py command that I ended up creating.

The image file needs to be within MEDIA_ROOT/photologue/photos to facilitate the import. The command is executed using ./manage.py addphoto 'photologue/photos/test.png'. Note that there is a --gallery option to add the image to a gallery, provided the gallery's slug.

from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
from photologue.models import Photo, Gallery

import os
from datetime import datetime
import pytz

class Command(BaseCommand):

    help = 'Adds a photo to Photologue.'

    def add_arguments(self, parser):
        parser.add_argument('imagefile',
                            type=str)
        parser.add_argument('--title',
                            type=str)
        parser.add_argument('--date_added',
                            type=str,
                            help="datetime string in 'YYYY-mm-dd HH:MM:SS' format [UTC]")
        parser.add_argument('--gallery',
                            type=str)

    def handle(self, *args, **options):
        imagefile = options['imagefile']
        base = os.path.basename(imagefile)

        if options['title']:
            title = options['title']
        else:
            title = os.path.splitext(base)[0]
        if options['date_added']:
            date_added = datetime.strptime(options['date_added'],'%Y-%m-%d %H:%M:%S').replace(tzinfo=pytz.UTC)
        else:
            date_added = timezone.now()

        try:
            p = Photo(image=imagefile, title=title, slug=title, date_added=date_added)
        except:
            raise CommandError('Photo "%s" could not be added' % base)
        p.save()
        self.stdout.write(self.style.SUCCESS('Successfully added photo "%s"' % p))

        if options['gallery']:
            try:
                g = Gallery.objects.get(slug=options['gallery'])
            except:
                raise CommandError('Gallery "%s" does not exist' % options['gallery'])
            p.galleries.add(g.pk)
            p.save()
            self.stdout.write(self.style.SUCCESS('Successfully added photo to gallery "%s"' % g))

Answer 2

On just one part of your question: the slug column is empty when the Photo is saved.

It should automatically be populated when the Photo is saved - as your copy-and-paste of the Photologue source code above if self.slug is None: self.slug = slugify(self.title) makes clear.

This suggests that the Photologue source code is not actually being called from your management command - you can check this by adding some quick debugging code to your local copy of the Photologue code, e.g. a print() statement in the save() method, and check if it is being run.

Answer 3

Edit/Update 2: I think I know why your slug field is empty. I think the issue is that self.slug isn't None, even though the field doesn't hold a string or holds an empty string (see this answer). So try updating if self.slug is None to this:

class Photo(ImageModel):
    ...
    def save(self, *args, **kwargs):
        # test if slug is not truthy, including empty string or None-ness
        if not self.slug:
            self.slug = slugify(self.title)
        super(Photo, self).save(*args, **kwargs)

Edit/Update 1: check out this answer. It is from Django 1.4 (ancient, I know), but it should solve your problem. If you copy or move the files you are adding to MEDIA_ROOT before creating the Photo instances, then you should be good to go. Here is an answer showing how to copy files in python. I suggest you change your custom command to this:

from django.core.management.base import BaseCommand, CommandError
from django.utils import timezone
from photologue.models import Photo, Gallery

# so you can access settings.MEDIA_ROOT
from django.conf import settings

# so you can copy file to MEDIA_ROOT if need be
from shutil import copyfile

import os
from datetime import datetime
import pytz

class Command(BaseCommand):

    help = 'Adds a photo to Photologue.'

    def add_arguments(self, parser):
        parser.add_argument('imagefile', type=str)

        # where the imagefile is currently located, assumes MEDIA_ROOT
        parser.add_argument('--media_source', type=str)
        parser.add_argument('--title', type=str)
        parser.add_argument('--date_added', type=str, help="datetime string in 'YYYY-mm-dd HH:MM:SS' format [UTC]")
        parser.add_argument('--gallery', type=str)

    def handle(self, *args, **options):

        # the path of the file relative to media_source
        imagefile = options['imagefile']

        # if file is not in media root, copy it to there
        if options['media_source']:
            media_source = os.path.realpath(options['media_source'])
            media_target = os.path.realpath(settings.MEDIA_ROOT)
            if media_source != media_target:
                copyfile(imagefile, os.path.join(media_target, imagefile)
        # note: if media_source was not provided, assume file is already in MEDIA_ROOT

        if options['title']:
            title = options['title']
        else:
            base = os.path.basename(imagefile)
            title = os.path.splitext(base)[0]
        if options['date_added']:
            date_added = datetime.strptime(options['date_added'],'%Y-%m-%d %H:%M:%S').replace(tzinfo=pytz.UTC)
        else:
            date_added = timezone.now()

        p = Photo(image=imagefile, title=title, date_added=date_added)
        p.save()

Now your imagefile is relative to the source of your media (../dataplots), gets copied to MEDIA_ROOT, and everything should work as planned. Here is what your command would look like

manage.py addphoto 'monitoring/test.png' --media_source='../dataplots'

Which should copy your dataplot into MEDIA_ROOT and then create the Photo entity as expected.

Original Answer:

Can you post what get_storage_path is in the following line:

class ImageModel(models.Model):
     image = models.ImageField(_('image'),
                          max_length=IMAGE_FIELD_MAX_LENGTH,
                          upload_to=get_storage_path)  # <-- here

This answer will be somewhat incomplete until I know what that is, but I think I see your issue. Look at the command you are running:

./manage.py addphoto '../dataplots/monitoring/test.png' --traceback

Your imagefile argument is ../dataplots/monitoring/test.png. If get_storage_path returns that same path, with the ../ at the start of the path, then you will be specifying an upload path that is not in your MEDIA_ROOT directory, but in its parent directory. I think it is trying to upload to MEDIA_ROOT/../dataplots/monitoring/test.png, as shown by the first SuspiciousFileOperation in your traceback:

# note: linebreaks and indentation added for readability
django.core.exceptions.SuspiciousFileOperation: 
    The joined path (/home/user/mysite/dataplots/monitoring/test.png) 
    is located outside of the base path component (/home/user/mysite/media)

So it is trying to upload to MEDIA_ROOT/imagefile, but imagefile begins with ../ which should not be allowed.

If this is indeed the problem (hard to say until you post the get_storage_path code), then there are many ways to solve the problem. Perhaps the quickest fix is just moving your dataplots directory into the same directory as your manage.py:

mv ../dataplots ./dataplots

That should solve your problem straightaway because you won't need the ../ anymore, but you probably don't want all of those dataplots in your project directory, so it is a quick but weak solution. The underlying problem is that the path of the source file and the path to which you are uploading the source file should not be expected to be equal. I think you should change your command arguments to include an image_source and image_destination, or you could include a source_media_root and imagefile where the ../ is part of source_media_root, and imagefile is relative to both source_media_root and the desired target location in MEDIA_ROOT... There are many solutions, but I can't quite provide the proper code for one until I know what get_storage_path is (I'm assuming it's a function that returns or can return the imagefile argument one way or another).