Resumable file upload in python

Question

I am trying to upload the file in python and i want to upload the file in resumable mode i.e when the internet connection resume , the file upload resume from the previous stage.

Is there any specific protocol that supports resumable file upload.

Thanks in advance

Answer 1

So what you need is to seek the file and send a REST command to tell the server to download from the right location.

Here is code that would try untill it finishes to upload with resume, also debug is on so you can follow:

#!/usr/bin/env python3
import ftplib
import os
import sys
import time
import socket

class FtpUploadTracker:
    sizeWritten = 0
    totalSize = 0.0
    lastShownPercent = 0

    def __init__(self, totalSize):
        self.totalSize = totalSize

    def handle(self, block):
        self.sizeWritten += 1024
        percentComplete = round((self.sizeWritten / self.totalSize) * 100)

        if (self.lastShownPercent != percentComplete):
            self.lastShownPercent = percentComplete
            print(str(percentComplete) + "% complete ramaing: " + str(self.totalSize - self.sizeWritten), flush=True)



if __name__ == "__main__":
    Server="servername.com"
    Username="username"
    Password="secret password"
    filename = "/path/to/folder"
    Directory="/path/on/server"

    tries = 0
    done = False

    print("Uploading " + str(filename) + " to " + str(Directory), flush=True)

    while tries < 50 and not done:
        try:
            tries += 1
            with ftplib.FTP(Server) as ftp:
                ftp.set_debuglevel(2)
                print("login", flush=True)
                ftp.login(Username, Password)
                # ftp.set_pasv(False)
                ftp.cwd(Directory)
                with open(filename, 'rb') as f:
                    totalSize = os.path.getsize(filename)
                    print('Total file size : ' + str(round(totalSize / 1024 / 1024 ,1)) + ' Mb', flush=True)
                    uploadTracker = FtpUploadTracker(int(totalSize))

                    # Get file size if exists
                    files_list = ftp.nlst()
                    print(files_list, flush=True)
                    if os.path.basename(filename) in files_list:
                        print("Resuming", flush=True)
                        ftp.voidcmd('TYPE I')
                        rest_pos = ftp.size(os.path.basename(filename))
                        f.seek(rest_pos, 0)
                        print("seek to " + str(rest_pos))
                        uploadTracker.sizeWritten = rest_pos
                        print(ftp.storbinary('STOR ' + os.path.basename(filename), f, blocksize=1024, callback=uploadTracker.handle, rest=rest_pos), flush=True)
                    else:
                        print(ftp.storbinary('STOR ' + os.path.basename(filename), f, 1024, uploadTracker.handle), flush=True)
                        done = True

        except (BrokenPipeError, ftplib.error_temp, socket.gaierror) as e:
            print(str(type(e)) + ": " + str(e))
            print("connection died, trying again")
            time.sleep(30)


    print("Done")

The magic line is:

print(ftp.storbinary('STOR ' + os.path.basename(filename), f, blocksize=1024, callback=uploadTracker.handle, rest=rest_pos), flush=True)

Which has rest=rest_pos.

If optional rest is given, a REST command is sent to the server, passing rest as an argument. rest is usually a byte offset into the requested file, telling the server to restart sending the file’s bytes at the requested offset, skipping over the initial bytes. Note however that RFC 959 requires only that rest be a string containing characters in the printable range from ASCII code 33 to ASCII code 126. The transfercmd() method, therefore, converts rest to a string, but no check is performed on the string’s contents. If the server does not recognize the REST command, an error_reply exception will be raised. If this happens, simply call transfercmd() without a rest argument

Source
Also some code taken from here

Answer 2

Excellent answer by GuySoft - it helped me a lot. I have had to slightly modify it as I never (so far) encountered the three exceptions his script is catching, but I experienced a lot of ConnectionResetError and socket.timeout errors on FTP uploads, so I added that. I also noticed that if I added a timeout of 60 seconds at ftp login, number of ConnectionResetErrors dropped significantly (but not all together). It was often happening that upload often got stuck at 100 % at the ftp.storbinary until socket.timeout, then tried 49 times and quit. I fixed that by comparing totalSize and rest_pos and exited when equal. So I have working solution now, bu I will try to figure out what is causing the socket timeouts. Interesting thing is that when I used Filezilla and even a PHP script, the file uploads to same FTP server were working without glitch.