AWS SDK generates wrong presigned url for reading an object

371 Views Asked by At

I am using S3 to store videos.
And now, I am using presigned urls to restrict access to them.
I am using these functions to generate the presigned url:

var getVideoReadSignedUrl = async function (url) {
  const key = awsS3Helpers.parseVideoKeyFromVideoUrl(url);
  return new Promise((resolve, reject) => {
    s3.getSignedUrl(
      "getObject",
      {
        Bucket: AWS_BUCKET_NAME,
        Key: key,
        Expires: 300,
      },
      (err, url) => {
        console.log(
          " ~ file: s3-config.js ~ line 77 ~ returnnewPromise ~ err",
          err
        );
        console.log(
          " ~ file: s3-config.js ~ line 77 ~ returnnewPromise ~ url",
          url
        );
        if (err) {
          reject(err);
        } else {
          resolve(url);
        }
      }
    );
  });
};

And this:

const parseVideoKeyFromVideoUrl = (object_url) => {
  const string_to_remove =
    "https://" + AWS_BUCKET_NAME + ".s3." + REGION + ".amazonaws.com/";
  const object_key = object_url.replace(string_to_remove, "");
  return object_key;
};

This is an example of a video url:

https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2F60e589xxxxxxxxx463c.mp4

So I call getVideoReadSignedUrl to get the signed url to give access to it:

getVideoReadSignedUrl(
  "https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2F60e589xxxxxxxxx463c.mp4"
);

parseVideoKeyFromVideoUrl correctly parses the key from the url:

videos%2F60e589xxxxxxxxx463c.mp4

BUT, this is what getVideoReadSignedUrl generates:

https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%252F60e589xxxxxxxxx463c.mp4?X-Amz-Algorithm=AWS4-xxx-xxxxx&X-Amz-Credential=AKIARxxxxxMVEWKUW%2F20221120%2Feu-xxxx-3%2Fs3%2Faws4_request&X-Amz-Date=20221120xxxx19Z&X-Amz-Expires=300&X-Amz-Signature=0203efcfaxxxxxxc53815746f75a357ff9d53fe581491d&X-Amz-SignedHeaders=host

When I open that url in the browser it tells me that the key does not exist:

<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>videos%2F60e589xxxxxxxxx463c.mp4</Key>

Even though in the error message the key is the same as in the original url.
But, I noticed a slight difference in the key between the original url and the presigned url:
Key in original url:

videos% |2F60| e589xxxxxxxxx463c.mp4

Key in signed url:

videos% |252F60| e589xxxxxxxxx463c.mp4

Not sure if this is causing the issue.


NOTE:
For the upload, I am using multipart upload and the reason the video url is structured like that is because of file_name here:

const completeMultiPartUpload = async (user_id, parts, upload_id) => {
  let file_name;
  file_name = `videos/${user_id}.mp4`;
  let params = {
    Bucket: AWS_BUCKET_NAME,
    Key: file_name,
    MultipartUpload: {
      Parts: parts,
    },
    UploadId: upload_id,
  };

  return data;
};

However, it should be stored like this:

/videos/e589xxxxxxxxx463c.mp4

and not like this:

/videos%2F60e589xxxxxxxxx463c.mp4

I am not sure why it replaces / with %2F60 and this may be what's causing the whole issue.


After more investigation, I found that completeMultiPartUpload returns this:

{
          Location:
            "https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4",
          Bucket: "lodeep-storage-3",
          Key: "videos/xxxxxxxxxxxxxxxxx.mp4",
          ETag: '"ee6xxxxxxxxxxxxf11-1"',
        }

And so the actualy object key is saved like this in S3:

"videos/xxxxxxxxxxxxxxxxx.mp4"

And this is the object url if I get it from AWS console:

https://BUCKET_NAME.s3.REGION.amazonaws.com/videos/xxxxxxxxxxxxxxxxx.mp4

But, in the database, this url gets saved like this since it's what completeMultiPartUpload function finally returns:

https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4

Notice how / is replaced with %2F.

So, when I am generating the signed url to read the video, the function that parses the key from the url parseVideoKeyFromVideoUrl, instead of getting the correct url in the S3:

https://BUCKET_NAME.s3.REGION.amazonaws.com/videos/xxxxxxxxxxxxxxxxx.mp4 It gets the one stored in the database: https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4

And so instead of returning this as a key:

/videos/xxxxxxxxxxxxxxxxx.mp4

It returns this:

/videos%2Fxxxxxxxxxxxxxxxxx.mp4

Which is an incorrect key. And so the signed url to read the video is incorrect. And I get this key doesn't exist error.

0

There are 0 best solutions below