I am trying to create an automated backup of an important directory, by first creating a zip file and then uploading the zipfile to an external storage bucket. There are quite a few subdirectories and the whole folder is approximately 80GB unzipped.
I looked around and found zipfile, alternatively using tarfile also appeared to be an option. However, for both the process throws an error around half way through - the below is the code for using zipfile, which throws an "OSError: [Errno 22] Invalid argument" error, around halfway through. I tried twice and it throws an error at different files, I also tried to just zip the files where it throws an error, but then everything works well.
I had the same problem using tarfile - I am at a loss of what is going on, my best guess is that the file is really large and it loads it into memory, at some point it runs out of memory and throws an error, however I cant figure out if this is even the process but just the only thing I could think of.
#File input/output
s_backup_output_fp=pathlib.Path(s_input_path+"_backup/full_backup.zip")
s_backup_input_fp=pathlib.Path(s_input_path+"folder to backup")
with zipfile.ZipFile(s_backup_output_fp, "w", zipfile.ZIP_DEFLATED) as zipf:
for fp in s_backup_input_fp.glob("**/*"):
zipf.write(fp, arcname=fp.relative_to(s_backup_input_fp))
Happy to provide more code/context, but I am pretty lost and has been working on this for a couple of days with no help.
"What would you like to get out of the answer": Ideally, an understanding of why my code throws an error and how to fix it so I know for this problem and future ones.