File upload to third party API with HTTP 307 Temporary Redirect in Flask

Question

I have a scenario where I have to upload a file from flask app to a third party API. I have wrapped all API requests in Flask to control API usage. For this I have redirected the traffic from main route towards the api wrapper route with http 307 status to preserve the request body and in the API wrapper I have used request to post into third party API endpoint.

The problem is only file < 100KB gets send through the redirection request, having a file larger than 100 KB gets somehow terminated in the sending phase.

Is there any limit in the 307 redirection and payload size?

I tried debugging by watching the network timing stack trace, from there it seems the request is dropped in the sending phase.

Main Blueprint

@main.route('/upload/',methods=['POST','GET'])  
def upload(): 
    #for ajax call
    if request.method == 'POST'
        return redirect(url_for('api.file_push'),code=307)
    else:
        return render_template('file-upload.html')

API Blueprint

@api.route('/upload/',methods=['POST'])
def file_push():
    upload_file = request.files['file']
    filename = urllib.parse.quote(upload_file.filename)
    toUpload = upload_file.read()
    result=requests.post(apiInterfaces.FILE_UPLOAD_INTERFACE+'/'+filename,files{'file':toUpload})
    return result

Yes, I can directly send post request to API endpoint from main route but I don't want to, it will destroy my system design and architecture.

Answer 1

I assume you're using Python, and possibly requests so this answer will be based on what I've learned figuring this out (debugging with a colleague). I filed a bug report with psf/requests. There is a related answer here which confirms my suspicions.

It seems that when you initiate a PUT request using requests (and urllib3), the entire request is sent before a response from the server is looked at, but some servers can send a HTTP 307 during this time. One of two things happen:

1. the server closes the connection by sending the response, even if the client has not finished sending the entire file. In this case, the client might see a closed connection and you won't have a response you can use for redirect (this happens with urllib3>1.26.10 (roughly)) but requests is not handling this situation correctly
2. the server sends the response and you re-upload the file to the second location (urllib3==1.26.3 has this behavior when using requests). Technically, there is a bug in urllib3 and it should have failed, but silently lets you upload...

However, it seems that if you are expecting a redirect, the most obvious solution might be to send a null byte via PUT first, get a valid response back for the new URL [don't follow redirects], and then use that response to do the PUT of the full file. With requests, it's probably something like

>>> import requests
>>> from io import BytesIO
>>> data = BytesIO(b'\x00')
>>> response = request.put(url, data=data, allow_redirects=False)
>>> request.put(response.headers['Location'], data=fp, allow_redirects=False)

and at that point, you'll be ok (assuming you only expect a single redirect here).