Python requests reading response while uploading request body

Question

I have a service to which I need to upload the content and the server starts sending the response after it gets certain amount of data, while my request body is still uploading.

headers = {'Content-Type': 'application/octet-stream',  'Expect': '100-continue', 'Connection' :'keep-alive'}
url = "https://MY_API_URL/WEBSERVICE"
response = requests.put(url, headers=headers,stream=True, data=data_gen(fh))
lines = response.iter_lines()
for line in lines:
    print line

data_gen is my generator function which takes a file handle of a very large file that yields 4KB per iteration.

My problem is that I dont get the "response" until the whole file uploads. Any ideas on how I can overcome this.

Answer 1

You cannot accomplish this with requests today. Requests (and the underlying libraries, including httplib/http.client [depending on your version of Python]) all send all of the data before they start reading the response.

One library that may be able to handle this (in fact, I'm fairly certain this should be doable with it) is treq. It uses Twisted which should give you ways to determine when data is received so all you should need to do is register a callback to start accessing that data.