Pre-signed URLS on Vultr Object Storage using boto3

Question

I have been working with Vultr for quite a while, and when I wanted to store some media files, I thought of AWS'S S3, and Vultr provides an S3 compatible service (Object Storage). I can use s3cmd CLI to play with the service, and they point to use boto3 for interacting with the S3 service. I wanted to have my objects have signed URLs, but I believe boto3 has amazonaws.com as the hostname as a constant somewhere in the code and that cannot be changed from configs as below in below:

import logging
from django.conf import settings
import boto3
from botocore.exceptions import ClientError
from premarket.models import PreMarket
from .models import SupervisorLogs


class Supervisor():

    def __init__(self) -> None:
        self.bucket_name = settings.BUCKET_NAME
        self.link_expiration = settings.ONE_WEEK
        self.queryset = PreMarket.objects.all()
        # Generate a presigned URL for the S3 object
        self.s3_configs_object = settings.AWS_S3_CREDS
        self.s3_client = boto3.client('s3', **self.s3_configs_object)

    def sign_objects(self):
        for obj in self.queryset:
            try:
                presigned_url = self.create_presigned_url(obj.video_url)
                obj.presigned_url = presigned_url
                obj.save()
            except Exception as e:
                self.supervisor_logs(level="ERROR", message=e, description=e)
                logging.error(e)
        self.supervisor_logs(level="COMPLETE",
                            message="Object URL signing has completed",
                            description="Object URL signing has completed")

    def create_presigned_url(self, object_name):
        """Generate a presigned URL to share an S3 object

        :param object_name: string
        :return: Presigned URL as string. If error, returns None.
        """

        try:
            response = self.s3_client.generate_presigned_url('get_object',
                                                            Params={
                                                                'Bucket': self.bucket_name,
                                                                'Key': object_name},
                                                            ExpiresIn=self.link_expiration)
        except ClientError as e:
            logging.error(e)
            self.supervisor_logs(level="ERROR", message=e, description=e)
            return None

        # The response contains the presigned URL
        return response

    def supervisor_logs(self, message: str = None, level: str = None, description: str = None) -> None:
        SupervisorLogs.objects.create(
            message=message, level=level, description=description)

AWS S3 credentials:

AWS_S3_CREDS = {
    'aws_access_key_id': AWS_ACCESS_KEY_ID,
    'aws_secret_access_key': AWS_SECRET_ACCESS_KEY,
    'region_name': 'ewr',
    'endpoint_url': AWS_BUCKET_HOSTNAME
}

But Vultr provides a different hostname as in:

ewr1.vultrobjects.com

Object example:

https://ewr1.vultrobjects.com/my-s3-bucket/basics/video.mp4

Trying to sign it with the code I wrote above produces the following (which is not working):

https://ewr1.vultrobjects.com/my-s3-bucket/https%3A//ewr1.vultrobjects.com/my-s3-bucket/basics/video.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AccessKeyHere%2F20210706%2Fewr%2Fs3%2Faws4_request&X-Amz-Date=20210706T133319Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=bb7da3183691052ea5b25218ab1fb876fec7053a2b045957fec12cd131393a19

Which has a few issues at first sight: the URL is invalid as is seems as if it is being concatenated, and even if I manually remove that just for testing purposes, it still would not work.

Vultr do not provide any documents regarding this as their support has informed me. At this point I am wondering how I can access the objects as they are (no pre-signed URLS), with just the Access key which would still not work.

I don't know what I am missing and I hope someone has some experience with this before as I could not find anything for the last couple days?

Thank you for your help.

Answer 1

The above code has you using the URL:

 presigned_url = self.create_presigned_url(obj.video_url)

Have you tried to use the s3 location to generate the URL? From the looks of it, the obj.video_url is https://ewr1...... where as the s3 URL would be s3:// and might be consumed better by the library you're using.

Also, before doing the entire loop for every URL, I'd try something manual like:

presigned_url = self.create_presigned_url('s3://ewr1.vultrobjects.com/my-s3-bucket/basics/video.mp4')

See if that gets you any joy.