How to resume from previous failed iteration while traversing through a big nested directory in Python?

47 Views Asked by At

I am currently using os.walk to navigate through all subfolders and files in a massive Network drive directory, However, Whenever my VPN disconnects, The for loop fails. Next day when I re-run my code, I would like to resume from the last file that was processed. What modifications should I make in my code below?

import os

directory = '//DirectoryName/FolderName'

for root, dirs, files in os.walk((os.path.normpath(directory)), topdown=False):
  for name in files:
        Source_File = os.path.join(root,name)
        #This loads the file to s3 bucket
        s3_client.upload_file(Source_File, bucket, Target_File)

The directory is really massive, Has hundreds of sub-folders, and thousands of files in total.

0

There are 0 best solutions below