OSError after SFTP successfully started with pysftp

Question

I am using put from pysftp to upload some files to a SFTP server.

I have a Python list filesToUpload_bordet and I am uploading each file x using a for loop within a context manager, as shown below:

# Enter the sftp server and perform the operations
with pysftp.Connection(host=hostname, 
                        username=username, 
                        password=password) as sftp:
    # Create and cd to the directory where the files will be uploaded
    try:
        sftp.mkdir(f'{lastRun}')
    except OSError:
        append(line=f"{timeStamp} run {lastRun}: {panel} WARNING: sftp directory /ngs/{lastRun} already exists, so not creating\n",logFile=logFile)
    with sftp.cd(f'/ngs/{lastRun}'):
        for x in filesToUpload_bordet:
            # put the vcf files
            sftp.put(x)

I know that the snipped above works because the upload successfully started. Though, after sometime I am getting the following error message on the Python console:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/nexus/databases/ngsRunStats_FK/postPipeline/scripts/newStrategy/STEPS_postPipeline_prod.py in <module>
----> 1 sftpUpload(panel)

/nexus/databases/ngsRunStats_FK/postPipeline/scripts/newStrategy/STEPS_postPipeline_prod.py in sftpUpload(panel)
     1212                 for x in filesToUpload_bordet:
     1213                     # put the vcf files
---> 1214                     sftp.put(x)
     1215         sftp.close() # this is probably not necessary as I am working on a context manager
     1216         # this message should be sent independently from the coverage check email

~/miniconda3/lib/python3.7/site-packages/pysftp/__init__.py in put(self, localpath, remotepath, callback, confirm, preserve_mtime)
    362 
    363         sftpattrs = self._sftp.put(localpath, remotepath, callback=callback,
--> 364                                    confirm=confirm)
    365         if preserve_mtime:
    366             self._sftp.utime(remotepath, times)

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_client.py in put(self, localpath, remotepath, callback, confirm)
    757         file_size = os.stat(localpath).st_size
    758         with open(localpath, "rb") as fl:
--> 759             return self.putfo(fl, remotepath, file_size, callback, confirm)
    760 
    761     def getfo(self, remotepath, fl, callback=None):

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_client.py in putfo(self, fl, remotepath, file_size, callback, confirm)
    715             fr.set_pipelined(True)
    716             size = self._transfer_with_callback(
--> 717                 reader=fl, writer=fr, file_size=file_size, callback=callback
    718             )
    719         if confirm:

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_client.py in _transfer_with_callback(self, reader, writer, file_size, callback)
    677         while True:
    678             data = reader.read(32768)
--> 679             writer.write(data)
    680             size += len(data)
    681             if len(data) == 0:

~/miniconda3/lib/python3.7/site-packages/paramiko/file.py in write(self, data)
    403             raise IOError("File not open for writing")
    404         if not (self._flags & self.FLAG_BUFFERED):
--> 405             self._write_all(data)
    406             return
    407         self._wbuffer.write(data)

~/miniconda3/lib/python3.7/site-packages/paramiko/file.py in _write_all(self, data)
    520         # a socket).
    521         while len(data) > 0:
--> 522             count = self._write(data)
    523             data = data[count:]
    524             if self._flags & self.FLAG_APPEND:

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_file.py in _write(self, data)
    206             while len(self._reqs):
    207                 req = self._reqs.popleft()
--> 208                 t, msg = self.sftp._read_response(req)
    209                 if t != CMD_STATUS:
    210                     raise SFTPError("Expected status")

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_client.py in _read_response(self, waitfor)
    863                 # synchronous
    864                 if t == CMD_STATUS:
--> 865                     self._convert_status(msg)
    866                 return t, msg
    867 

~/miniconda3/lib/python3.7/site-packages/paramiko/sftp_client.py in _convert_status(self, msg)
    896             raise IOError(errno.EACCES, text)
    897         else:
--> 898             raise IOError(text)
    899 
    900     def _adjust_cwd(self, path):

OSError: Failure

Can this be anything other than a time-out issue? I see that my first file was successfully uploaded at 09:08am and I got the error message at 11:49am.

Answer 1

The "Failure" is an error message for SFTP error code 4, returned by the OpenSSH SFTP server for various problems, for which there's no more specific code in the SFTP protocol version 3. While the server should at least return a specific plain-text error message, it fails to do so.

Common reasons you may get the generic "Failure" error message, while uploading are:

• Uploading a file to a full filesystem (HDD).
• Exceeding a user disk quota.

For details, see SFTP Status/Error Code 4 (Failure).