AWS S3 POST File from IOS client Using AFNetworking

Question

I am hoping, through this post, not just to get an answer but to help many others struggling with using AWS secure POST policy infrastructure. Code is shown, after the explanation.

We use an EC2 Ubuntu 20.2 instance running Django as server. Using boto3 (s3_client.generate_presigned_post) we create a RetrieveAPIView using rest_framework. This Generic View accepts 3 GET parameters - file_name, meta_uuid & mime_type - and generates the POST signature and policy. This view returns the "fields" dictionary, url, etc. in the form of JSON. Boto3 ignores the value entered for "x-amz-date" and generates its own.

Using Postman we can successfully retrieve a full policy from the Ubuntu server. Then, after copying/pasting the values for the returned fields we use Postman to post the file to AWS using form-data in the body. The file uploads fine and we get the HTTP 204 No Content and the other fields in the Header. If we make a small change to one of the fields during the post to AWS S3, the server comes back with HTTP 403 Forbidden & the code "SignatureDoesNotMatch", just what one would expect.

BUT, when attempting to run from EITHER the Xcode simulator or from an actual device using AFNetworking's [AFHTTPSessionManager uploadTaskWithRequest: fromFile: progress: completionHandler:] we get back from AWS S3 - HTTP 400 Bad Request and the code "IncompleteBody" in the xml. AWS documentation suggests that the problem is that the content length measured at S3 not consistent with the Content-Length being sent in the header.

The file size, when measured using iOS FileManger is 111251 . AFNetworking outbound Header is Content-Length = 113071. That makes sense when one counts the length of the encoded policy, signature, other fields, the boundary value, etc.

Any hints on what to look for would be greatly appreciated. Does AWS only look at the file size when calculating Content-Length? Does this mean that the policy is OK, or does AWS S3 first just do some rudimentary header checking before looking at the encoded policy and signature?

IOS using AFNetworking 3.0 (CocoaPod), Xcode 13.1.

Thanks,

Eric NB: Policy encodings were fudged so as not to expose our MY_AWS_ASSIGNED_S3_KEY_ID

Code follows and hoping it helps many other dealing with this same issue:

Server side Django:

    def retrieve(self, request, *args, **kwargs):
        s3_client = boto3.client('s3',  aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
                                        aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY'),
                                        region_name=AWS_REGION_NAME,
                                        config=config.Config(signature_version='s3v4'))
        t = datetime.datetime.utcnow()
        amz_date = t.strftime('%Y%m%d')  # Date w/o time, used in credential scope
        amz_date_z = amz_date + 'T000000Z'
        amz_cred = os.getenv('AWS_ACCESS_KEY_ID') + '/' + amz_date + '/' + AWS_REGION_NAME + '/' + 's3/aws4_request'
        cont_type = request.GET.get('mime_type', None)
        meta_uuid = request.GET.get('meta_uuid', None)
        fields_dict = {
            'acl': 'authenticated-read',
            'bucket': AWS_STORAGE_BUCKET_NAME,
            'x-amz-algorithm': 'AWS4-HMAC-SHA256',
            'x-amz-meta-uuid': meta_uuid,
            'x-amz-credential': amz_cred,
            'x-amz-date': amz_date_z,
            'content-type': cont_type
        }
        cond_list = [
            {'content-type': cont_type},
            {'bucket': AWS_STORAGE_BUCKET_NAME},
            {'x-amz-meta-uuid': meta_uuid},
            {'acl': 'authenticated-read'},
        ]
        ret_dict = s3_client.generate_presigned_post(
            Bucket=AWS_STORAGE_BUCKET_NAME,
            Key=PRIVATE_MEDIA_LOCATION + request.GET.get('file_name', None),
            Fields=fields_dict,
            Conditions=cond_list,
            ExpiresIn=3600
        )
        print("ret_dict =")
        print(ret_dict)
        return Response({'status': 'success', 'data': ret_dict}, status=status.HTTP_202_ACCEPTED)

Returned JSON from server to Postman:

{
    "status": "success",
    "data": {
        "url": "https://aa-dev-media.s3.amazonaws.com/",
        "fields": {
            "acl": "authenticated-read",
            "bucket": "aa-dev-media",
            "x-amz-algorithm": "AWS4-HMAC-SHA256",
            "x-amz-meta-uuid": "some_random_string",
            "x-amz-credential": "**MY_AWS_ASSIGNED_S3_KEY_ID**/20211107/us-west-2/s3/aws4_request",
            "x-amz-date": "20211107T092556Z",
            "content-type": "image/jpeg",
            "key": "media/krTkQskg.jpg",
            "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMTEtMDdUMTA6MjU6NTZaIiwgImNvbmRpdGlvbnMiOiBbeyJjb250ZW50LXR5cGUiOiAiaW1hZ2UvanBlZyJ9LCB7ImJ1Y2tldCI6ICJhYS1kZXYtbWVkaWEifSwgeyJ4LWFtei1tZXRhLXV1aWQiOiAic29tZV9yYW5kb21fc3RyaW5nIn0sIHsiYWNsIjogImF1dGhlbnRpY2F0ZWQtcmVhZCJ9LCB7ImJ1Y2tldCI6ICJhYS1kZXYtbWVkaWTTTSwgeyJrZXkiOiAibWVkaWEva3JUa1Fza2cuanBnIn0sIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwgeyJ4LWFtei1jcmVkZW50aWFsIjogIkFLSUE2UVZNTFlWNVRSUk9YQjVZLzIwMjExMTA3L3VzLXdlc3QtMi9zMy9hd3M0X3JlcXVlc3QifSwgeyJ4LWFtei1kYXRlIjogIjIwMjExMTA3VDA5MjU1NloifV19",
            "x-amz-signature": "8fcec036a12ae2ab9212133c5c7ce275e91961ab9486936a47ce0e6bb869e6ce"
        }
    }
}

Objective C code in App:

-   (void)uploadCurrentMediaFileToS3ForAPI:(APIType)api usingParameters:(NSDictionary *)pDict
{
    NSLog(@"%s fileURL = %@",__FUNCTION__,currentMFO.file_url);
    NSLog(@"%s fName = %@",__FUNCTION__,currentMFO.file_name);
    NSLog(@"%s mType = %@",__FUNCTION__,currentMFO.mime_mfo.mime_type);
    NSLog(@"%s httpPath = %@",__FUNCTION__,httpPath);
    NSLog(@"%s pDict = %@",__FUNCTION__,pDict);
    NSLog(@"%s currentMFO.file_size_bytes = %lld",__FUNCTION__,currentMFO.file_size_bytes);

    NSError *reqError = nil;
    AFHTTPRequestSerializer *reqSerial  = [AFHTTPRequestSerializer serializer];
    NSMutableURLRequest *req = [reqSerial multipartFormRequestWithMethod:kHTTPPOST URLString:httpPath parameters:nil constructingBodyWithBlock:^(id<AFMultipartFormData>  _Nonnull formData)
    {
        
        NSArray *paramKeys  =   [pDict allKeys];
        for (NSString *key in paramKeys)
        {
            NSString *paramValue =  [pDict objectForKey:key];
            NSData *valData     =   [paramValue dataUsingEncoding:NSUTF8StringEncoding];
            NSLog(@"%s key = %@, value = %@ ",__FUNCTION__,key, paramValue);
            [formData   appendPartWithFormData:valData name:key];
        }
        
        BOOL fileFormSuccess    =   [formData appendPartWithFileURL:self->currentMFO.file_url name:@"file" fileName:self->currentMFO.file_name mimeType:self->currentMFO.mime_mfo.mime_type error:nil];
        NSLog(@"%s fileFormSuccess = %i",__FUNCTION__,fileFormSuccess);
    } error:&reqError];
    if (reqError) [self incrementErrorCountForAPI:api statusCode:100 locDesc:reqError.localizedDescription andReason:reqError.localizedFailureReason];
    [req addValue:@"*/*" forHTTPHeaderField:@"Accept"];
    [req addValue:@"500" forHTTPHeaderField:@"Keep-Alive"];
    NSLog(@"%s after req.allHTTPHeaderFields = %@",__FUNCTION__,req.allHTTPHeaderFields);
    NSLog(@"%s req.URL = %@",__FUNCTION__,req.URL);

    AFHTTPSessionManager *manager   =   [[AFHTTPSessionManager alloc] initWithSessionConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]];
    manager.responseSerializer      =   [AFHTTPResponseSerializer   serializer];
    NSURLSessionUploadTask *uploadTask  =   [manager uploadTaskWithRequest:req fromFile:currentMFO.file_url progress:nil completionHandler:^(NSURLResponse * _Nonnull response, id  _Nullable responseObject, NSError * _Nullable error)
    {
        NSHTTPURLResponse *httpResp =   (NSHTTPURLResponse *)response;
        NSLog(@"%s httpResp.allHeaderFields - %@",__FUNCTION__,httpResp.allHeaderFields);
        NSLog(@"%s httpResp.MIMEType - %@",__FUNCTION__,httpResp.MIMEType);
        NSLog(@"%s httpResp.statusCode - %ld",__FUNCTION__,(long)httpResp.statusCode);
        if (responseObject)
        {
            NSString *respString    =   [[NSString alloc] initWithData:responseObject encoding:NSUTF8StringEncoding];
            NSLog(@"%s respString - %@",__FUNCTION__,respString);
        }
    }];
    [uploadTask resume];
}

And here are the logs from the Xcode debugger:

fileURL = file:///var/mobile/Containers/Data/Application/8FF62939-0E50-4582-B242-01472D1C75D3/Library/UserMedia/horiz_windlass_photo.png
fName = horiz_windlass_photo.png
mType = image/png
httpPath = https://aa-dev-media.s3.amazonaws.com/
pDict = {
    acl = "authenticated-read";
    bucket = "aa-dev-media";
    "content-type" = "image/png";
    key = "media/horiz_windlass_photo.png";
    policy = "eyJleHBpcmF0aW9uIjogIjIwMjEtMTEtMDdUMTE6MjM6MDNaIiwgImNvbmRpdGlvbnMiOiBbeyJjb250ZW50LXR5cGUiOiAiaW1hZ2UvcG5nIn0sIHsiYnVja2V0IjogImFhLWRldi1tZWRpYSJ9LCB7IngtYW16LW1ldGEtdXVpZCI6ICJob3Jpel93aW5kbGFzc19waG90by5wbmcifSwgeyJhY2wiOiAiYXV0aGVudGljYXRlZC1yZWFkIn0sIHsiYnVja2V0IjogImFhLWRldi1tZWRpYSJ9LCB7ImtleSI6ICJtZWRpYS9ob3Jpel93aW5kbGFzc19waG90by5wbmcifSwgeyJ4LXXXXi1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQTZRVk1MWVY1VFJST1hCNVkvMjAyMTExMDcvdXMtd2VzdC0yL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTExMDdUMTAyMzAzWiJ9XX0=";
    "x-amz-algorithm" = "AWS4-HMAC-SHA256";
    "x-amz-credential" = "**MY_AWS_ASSIGNED_S3_KEY_ID**/20211107/us-west-2/s3/aws4_request";
    "x-amz-date" = 20211107T102303Z;
    "x-amz-meta-uuid" = "horiz_windlass_photo.png";
    "x-amz-signature" = 41677f9254e324553a429bbcc3dcf5bd0f5df4af1c25fcaf51ce088a2a5dd032;
}
currentMFO.file_size_bytes = 111251
key = bucket, value = aa-dev-media
key = content-type, value = image/png
key = policy, value = eyJleHBpcmF0aW9uIjogIjIwMjEtMTEtMDdUMTE6MjM6MDNaIiwgImNvbmRpdGlvbnMiOiBbeyJjb250ZW50LXR5cGUiOiAiaW1hZ2UvcG5nIn0sIHsiYnVja2V0IjogImFhLWRldi1tZWRpYSJ9LCB7IngtYW16LW1ldGEtdXVpZCI6ICJob3Jpel93aW5kbGFzc19waG90by5wbmcifSwgeyJhY2wiOiAiYXV0aGVudGljYXRlZC1yZWFkIn0sIHsiYnVja2V0IjogImFhLWRldi1tZWRpYSJ9LCB7ImtleSI6ICJtZWRpYS9ob3Jpel93aW5kbGFzc19waG90by5wbmcifSwgeyJ4LXXXXi1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQTZRVk1MWVY1VFJST1hCNVkvMjAyMTExMDcvdXMtd2VzdC0yL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTExMDdUMTAyMzAzWiJ9XX0=
key = x-amz-signature, value = 41677f9254e324553a429bbcc3dcf5bd0f5df4af1c25fcaf51ce088a2a5dd032
key = x-amz-algorithm, value = AWS4-HMAC-SHA256
key = acl, value = authenticated-read
key = x-amz-meta-uuid, value = horiz_windlass_photo.png
key = key, value = media/horiz_windlass_photo.png
key = x-amz-credential, value = **MY_AWS_ASSIGNED_S3_KEY_ID**/20211107/us-west-2/s3/aws4_request
key = x-amz-date, value = 20211107T102303Z
after formData = <AFStreamingMultipartFormData: 0x282c6e0d0>
fileFormSuccess = 1
req.allHTTPHeaderFields = {
    User-Agent = "AnchorAway/1.0 (iPhone; iOS 15.0.2; Scale/3.00)”,
    Accept-Language = "en-US;q=1”,
    Content-Type =  "multipart/form-data; boundary=Boundary+5D68022C08D21CC8”,
    Content-Length = 113071,
    Accept =  “*/*”,
    Keep-Alive = "500"
]
req.HTTPBody = (null)
eq.URL = https://aa-dev-media.s3.amazonaws.com/}
httpResp.allHeaderFields - {
    Server = AmazonS3
    Content-Type = "application/xml"
    Transfer-Encoding = "Identity"
    x-amz-request-id = "2BXQ22BB603CA92E"
    Date = "Sun, 07 Nov 2021 10:23:03 GMT"
    x-amz-id-2 = "tmvZxT96gJvL4SwxTdlHhkh3ZrGj1vmiG4JO1MQPyJc3bYlXMpofLxfImLuhBZEddSAS7nUdAzc="
    Connection = close
}
httpResp.MIMEType - application/xml
httpResp.statusCode - 400
respString - <?xml version="1.0" encoding="UTF-8"?><Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly</Message> <RequestId>2BXQ22BB603CA92E<RequestId><HostId>tmvZxT96gJvL4SwxTdlHhkh3ZrGj1vmiG4JO1MQPyJc3bYlXMpofLxfImLuhBZEddSAS7nUdAzc=</HostId></Error>

Answer 1

I found the solutions and willing to share it. If you like or use it, please vote 'yes' so perhaps I get enough visibility so that next time someone posts advise... Approach should also work just fine in Swift...

1. Using AFNetworking's uploadTaskWithRequest:fromFile: in conjunction with its multipartFormRequestWithMethod:URLString:parameters:constructingBodyWithBlock:error: ------ was a bad idea.

AFNetworking's is a subclass of Apple's uploadTaskWithRequest:fromFile: . Apple Documentation States "The body stream and body data in this request object are ignored." Hence, anything formed during construction of the Body was ignored subsequently by uploadTask. AWS never saw a multi-form body or file.

2. AWS's S3 Response Error Codes cannot be taken literally

That is not to say that they aren't useful, because they are. But, often they are "red herrings" for the real issue, which requires a lot of brute force digging. What saved me on this was using WireShark to examine the packets in the request and response --- extremely helpful - https://www.wireshark.org I highly recommend using this tool.

3. Manually construct your own form-data body as string, convert to NSData, add data from image file, append with form-data suffix and save to temporary file

You might find this code useful:

NSString * boundaryString       =   [NSString stringWithFormat:@"------XY%@",SOME_RANDOM_STRING];
NSMutableString *bodyString =   [NSMutableString stringWithFormat:@"--%@\r\n",boundaryString];
for (NSString *key in paramKeys)
{
    [bodyString appendFormat:@"Content-Disposition: form-data; name=\"%@\"\r\n\r\n",key];
    [bodyString appendFormat:@"%@",[paramDict objectForKey:key]];
    [bodyString appendFormat:@"\r\n--%@\r\n", boundaryString];
}
[bodyString appendFormat:@"Content-Disposition: form-data; name=\"%@\"; filename=\"%@\"\r\n",@"file",currentMFO.file_name];
[bodyString appendFormat:@"Content-Type: %@\r\n\r\n",currentMFO.mime_mfo.mime_type];
NSMutableData *postData =   [NSMutableData dataWithData:[bodyString dataUsingEncoding:NSUTF8StringEncoding]];
NSData *fileData        =   [NSData dataWithContentsOfFile:IMAGE_FILE_URL];
[postData appendData:fileData];
NSString *suffixString  =   [NSString stringWithFormat:@"\r\n--%@--\r\n",boundartString];
NSData *suffixData      =   [NSData dataWithData:[suffixString dataUsingEncoding:NSUTF8StringEncoding]];
[postData appendData:suffixData];
[[NSFileManager defaultManager] createFileAtPath:PATH_FOR_TEMP_FILE contents:postData attributes:nil];

4. Finally, passing the temporary file url to AFNetworking's uploadTaskWithRequest:fromFile: should work just fine.