Why to use iter_content and chunk_size in python requests

Question

Why should I use iter_content and specially I'm really confused with the purpose using of chunk_size , as I have tried using it and in every way the file seems to be saved after downloading successfully.

g = requests.get(url, stream=True)

with open('c:/users/andriken/desktop/tiger.jpg', 'wb') as sav:
    for chunk in g.iter_content(chunk_size=1000000):
        print (chunk)
        sav.write(chunk)

Help me understand the use of iter_content and what will happen as you see I am using 1000000 bytes as chunk_size, what is the purpose exactly and results?

Answer 1

This is to prevent loading the entire response into memory at once (it also allows you to implement some concurrency while you stream the response so that you can do work while waiting for request to finish).

The purpose of setting streaming request is usually for media. Like try to download a 500 MB .mp4 file using requests, you want to stream the response (and write the stream in chunks of chunk_size) instead of waiting for all 500mb to be loaded into python at once.

If you want to implement any UI feedback (such as download progress like "downloaded <chunk_size> bytes..."), you will need to stream and chunk. If your response contains a Content-Size header, you can calculate % completion on every chunk you save too.

Answer 2

From the documentations chunk_size is size of data, that app will be reading in memory when stream=True.

For example, if the size of the response is 1000 and chunk_size set to 100, we split the response into ten chunks.

Answer 3

In situations when data is delivered without a content-length header, using HTTP1.1 chunked transfer encoding (CTE) mode or HTTP2/3 data frames, where minimal latency is required it can be useful to deal with each HTTP chunk as it arrives, as opposed to waiting till the buffer hits a specific size.

This can be achieved by setting chunk_size = None.