I am using S3 to store videos.
And now, I am using presigned urls to restrict access to them.
I am using these functions to generate the presigned url:
var getVideoReadSignedUrl = async function (url) {
const key = awsS3Helpers.parseVideoKeyFromVideoUrl(url);
return new Promise((resolve, reject) => {
s3.getSignedUrl(
"getObject",
{
Bucket: AWS_BUCKET_NAME,
Key: key,
Expires: 300,
},
(err, url) => {
console.log(
" ~ file: s3-config.js ~ line 77 ~ returnnewPromise ~ err",
err
);
console.log(
" ~ file: s3-config.js ~ line 77 ~ returnnewPromise ~ url",
url
);
if (err) {
reject(err);
} else {
resolve(url);
}
}
);
});
};
And this:
const parseVideoKeyFromVideoUrl = (object_url) => {
const string_to_remove =
"https://" + AWS_BUCKET_NAME + ".s3." + REGION + ".amazonaws.com/";
const object_key = object_url.replace(string_to_remove, "");
return object_key;
};
This is an example of a video url:
https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2F60e589xxxxxxxxx463c.mp4
So I call getVideoReadSignedUrl
to get the signed url to give access to it:
getVideoReadSignedUrl(
"https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2F60e589xxxxxxxxx463c.mp4"
);
parseVideoKeyFromVideoUrl
correctly parses the key from the url:
videos%2F60e589xxxxxxxxx463c.mp4
BUT, this is what getVideoReadSignedUrl
generates:
When I open that url in the browser it tells me that the key does not exist:
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>videos%2F60e589xxxxxxxxx463c.mp4</Key>
Even though in the error message the key is the same as in the original url.
But, I noticed a slight difference in the key between the original url and the presigned url:
Key in original url:
videos% |2F60| e589xxxxxxxxx463c.mp4
Key in signed url:
videos% |252F60| e589xxxxxxxxx463c.mp4
Not sure if this is causing the issue.
NOTE:
For the upload, I am using multipart upload and the reason the video url is structured like that is because of file_name
here:
const completeMultiPartUpload = async (user_id, parts, upload_id) => {
let file_name;
file_name = `videos/${user_id}.mp4`;
let params = {
Bucket: AWS_BUCKET_NAME,
Key: file_name,
MultipartUpload: {
Parts: parts,
},
UploadId: upload_id,
};
return data;
};
However, it should be stored like this:
/videos/e589xxxxxxxxx463c.mp4
and not like this:
/videos%2F60e589xxxxxxxxx463c.mp4
I am not sure why it replaces / with %2F60 and this may be what's causing the whole issue.
After more investigation, I found that completeMultiPartUpload
returns this:
{
Location:
"https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4",
Bucket: "lodeep-storage-3",
Key: "videos/xxxxxxxxxxxxxxxxx.mp4",
ETag: '"ee6xxxxxxxxxxxxf11-1"',
}
And so the actualy object key is saved like this in S3:
"videos/xxxxxxxxxxxxxxxxx.mp4"
And this is the object url if I get it from AWS console:
https://BUCKET_NAME.s3.REGION.amazonaws.com/videos/xxxxxxxxxxxxxxxxx.mp4
But, in the database, this url gets saved like this since it's what completeMultiPartUpload
function finally returns:
https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4
Notice how / is replaced with %2F.
So, when I am generating the signed url to read the video, the function that parses the key from the url parseVideoKeyFromVideoUrl
, instead of getting the correct url in the S3:
https://BUCKET_NAME.s3.REGION.amazonaws.com/videos/xxxxxxxxxxxxxxxxx.mp4 It gets the one stored in the database: https://BUCKET_NAME.s3.REGION.amazonaws.com/videos%2Fxxxxxxxxxxxxxxxxx.mp4
And so instead of returning this as a key:
/videos/xxxxxxxxxxxxxxxxx.mp4
It returns this:
/videos%2Fxxxxxxxxxxxxxxxxx.mp4
Which is an incorrect key. And so the signed url to read the video is incorrect. And I get this key doesn't exist error.