How to stream more than 4GB zip files while generating the zip file in python

51 Views Asked by At

I have n number of audio files in s3. If I make a django api request, it should get all the audio files from s3 and generate a zip file and download it as zip file. While generating the zip file, it should start streaming the zip file foe download. So far i have generated the zip file but at the same time if i stream it, it downloads a corrupted zip file. I have used aiozipstream library to achieve it but it has limit of upto 4GB zip file. below is the sample code with different approach

def generate_stream_zip(urls):

    zip_buffer = BytesIO()
    s3_client = boto3.client(
        's3',
        aws_access_key_id = settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key= settings.AWS_SECRET_ACCESS_KEY
    )
    bucket = settings.AWS_STORAGE_BUCKET_NAME

    with ZipFile(zip_buffer, 'w', ZIP_DEFLATED, allowZip64=True) as zip_file:
        for url, name, key in urls:
            key = f'media/{key}'
            obj = s3_client.get_object(Bucket=bucket, Key=key)
            zip_file.writestr(name, obj['Body'].read())
            yield zip_file.read(name)


response = StreamingHttpResponse(generate_stream_zip(urls), content_type='application/zip')
response['Content-Disposition'] = 'attachment; filename=mibe_playlist.zip'
return response

0

There are 0 best solutions below