FtpWebRequest throws the following exception after repeated calls: "The underlying connection was closed: An unexpected error occurred on a receive."

Question

My goal is to write a program that would download a directory from an ftp server with all the files and nested directories inside of it. So to do this I need two functions: 1) list all files/directories within a directory and 2) download a specific file. Then I can traverse through a directory recursively and recreate it on my local computer.

So let me show you the code that I'm using.

1) For directory traversal unfortunately I couldn't find a definitive way to do it reliably with FtpWebRequest. So in my implementation I'm using two ftp commands ListDirectory and ListDirectoryDetails. The first one gives me file/directory names and the second one is used to determine whether it's a file or a directory. (Unfortunately I couldn't find a reliable way to parse the output returned by ListDirectoryDetails -- everyone seems to be using their own regex for that, and to make matters worse it seems like different ftp servers may report it in a different way as well.)

//Example of strURI = "ftp://server123.domain.com/%2F/public_html"
FtpWebRequest ftpRequest1 = (FtpWebRequest)WebRequest.Create(strURI);
ftpRequest1.EnableSsl = true;   //Use TLS!
ftpRequest1.Credentials = credentials;
ftpRequest1.KeepAlive = kbKeepAlive;
ftpRequest1.Timeout = knFtpTimeout;

ftpRequest1.Method = WebRequestMethods.Ftp.ListDirectoryDetails;

using (FtpWebResponse response1 = (FtpWebResponse)ftpRequest1.GetResponse())
{
    FtpWebRequest ftpRequest2 = (FtpWebRequest)WebRequest.Create(strURI);
    ftpRequest2.EnableSsl = true;   //Use TLS!
    ftpRequest2.Credentials = credentials;
    ftpRequest2.KeepAlive = kbKeepAlive;
    ftpRequest2.Timeout = knFtpTimeout;

    ftpRequest2.Method = WebRequestMethods.Ftp.ListDirectory;

    using (StreamReader streamReader = new StreamReader(response1.GetResponseStream()))
    {
        List<string> arrFullDirList = new List<string>();

        for (; ; )
        {
            string line = streamReader.ReadLine();
            if (string.IsNullOrEmpty(line))
                break;

            arrFullDirList.Add(line);
        }

        using (FtpWebResponse response2 = (FtpWebResponse)ftpRequest2.GetResponse())
        {
            using (StreamReader streamReader2 = new StreamReader(response2.GetResponseStream()))
            {
                List<string> arrDirNamesList = new List<string>();

                for (; ; )
                {
                    string line = streamReader2.ReadLine();
                    if (string.IsNullOrEmpty(line))
                        break;

                    arrDirNamesList.Add(line);
                }

                //Then analyze two arrays
                if (arrFullDirList.Count == arrDirNamesList.Count)
                {
                    //If the first char in `arrFullDirList` item is 'd' then
                    //it's a directory. If it's '-' then it's a file...
                }
            }

            response2.Close();
        }
    }

    response1.Close();
}

2) And for file downloads I use this code:

//Query size of the file to be downloaded
//Example of strURI = "ftp://server123.domain.com/%2F/public_html/.htaccess"
FtpWebRequest requestSz = (FtpWebRequest)WebRequest.Create(strURI);
requestSz.EnableSsl = true; //Use TLS!
requestSz.Credentials = credentials;
requestSz.UseBinary = true;
requestSz.Method = WebRequestMethods.Ftp.GetFileSize;
requestSz.Timeout = knFtpTimeout;

using (FtpWebResponse responseSize = (FtpWebResponse)requestSz.GetResponse())
{
    //File size
    long uiFileSize = responseSize.ContentLength;

    //Download file request
    FtpWebRequest request = (FtpWebRequest)WebRequest.Create(strURI);
    request.EnableSsl = true;   //Use TLS!
    request.Credentials = credentials;
    request.UseBinary = true;
    request.Timeout = knFtpTimeout;

    request.Method = WebRequestMethods.Ftp.DownloadFile;

    using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
    {
        using (Stream ftpStream = response.GetResponseStream())
        {
            const int kBufferLength = 1024;
            byte[] buffer = new byte[kBufferLength];

            using (FileStream streamFile = File.Create(strLocalFilePath))
            {
                for (long uiProcessedSize = 0; ; )
                {
                    int ncbRead = ftpStream.Read(buffer, 0, kBufferLength);
                    if (ncbRead == 0)
                    {
                        break;
                    }

                    streamFile.Write(buffer, 0, ncbRead);

                    uiProcessedSize += ncbRead;

                    double fProgress = (double)uiProcessedSize / (double)uiFileSize;
                    fProgress *= 100.0;

                    //Show download progress for a user ...
                }
            }
        }

        response.Close();
    }

    responseSize.Close();
}

and lastly I use the following globals:

    const bool kbKeepAlive = true;
    const int knFtpTimeout = -1;

    NetworkCredential credentials = new NetworkCredential("user_name", "password");

So I'm using this approach to try to download my small web site files from a (paid) shared web hosting server. This works in a very strange way:

• If I set kbKeepAlive = true the download works much faster, but then at some predictable point one of the FtpWebResponse calls will throw this exception:

The underlying connection was closed: An unexpected error occurred on a receive.

• If I set kbKeepAlive = false the download is slower and progresses much further than the example above, but at some point it will too throw the same exception.

This process is repeatable and it feels almost like there's a counter, or some resource that is being depleted that eventually runs out that causes this error.

So I did some search. There're quite a few hits for this exception, but unfortunately none of the fixes suggested seem to work for me:

a) Someone suggested increasing the timeout. Well, I totally removed it with my knFtpTimeout = -1 and it made no difference whatsoever.

b) Then someone suggested to enable network tracing and check the log. So I did that as well. Here's what I got in the log. I'm not sure if it answers anything:

(The log itself is very large, but here's the end of it. I edited out the actual ftp server URL.)

[Subject]
  CN=*.domain.com, OU=PositiveSSL Wildcard, OU=Domain Control Validated
  Simple Name: *.domain.com
  DNS Name: domain.com

[Issuer]
  CN=COMODO RSA Domain Validation Secure Server CA, O=COMODO CA Limited, L=Salford, S=Greater Manchester, C=GB
  Simple Name: COMODO RSA Domain Validation Secure Server CA
  DNS Name: COMODO RSA Domain Validation Secure Server CA

[Serial Number]
  00F61110ACEE4BF307C442973F3B842A2E

[Not Before]
  2/3/2015 4:00:00 PM

[Not After]
  3/7/2018 3:59:59 PM

[Thumbprint]
  F429DCB7B8181F0236432890C3E10D99719AC698

[Signature Algorithm]
  sha256RSA(1.2.840.113549.1.1.11)

[Public Key]
  Algorithm: RSA
  Length: 2048
  Key Blob: 30 82 01 0a 02 82 01 01 00 d1 68 8c 67 75 9a f2 93 b1 25 95 2d 43 32 d6 83 18 07 09 ba 1a 2a d3 b3 b6 09 75 eb 92 05 9d 41 ab 38 ad a7 af 2a d1 5e f4 02 21 b0 d5 8b fe 54 17 da 90 2f c2 06 c7 6a 8b 57 fb 1b f9 4d 25 c4 9d b0 7c 31 94 19 ee 27 9c 81 ed b1 01 ba 7e 06 0f af d8 9a 81 94 25 ....
System.Net Information: 0 : [12064] SecureChannel#32368095 - Remote certificate was verified as valid by the user.
System.Net Information: 0 : [12064] ProcessAuthentication(Protocol=Tls, Cipher=Aes128 128 bit strength, Hash=Sha1 160 bit strength, Key Exchange=44550 256 bit strength).
System.Net Error: 0 : [12064] Decrypt failed with error 0X90317.
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Received response [226-File successfully transferred
226 0.000 seconds (measured here), 0.58 Mbytes per second]
System.Net Information: 0 : [12064] FtpWebRequest#21940722::(Releasing FTP connection#11318800.)
System.Net Information: 0 : [12064] FtpWebRequest#41130254::.ctor(ftp://server123.domain.com///public_html/ctc)
System.Net Information: 0 : [12064] FtpWebRequest#41130254::GetResponse(Method=LIST.)
System.Net Information: 0 : [12064] Associating FtpWebRequest#41130254 with FtpControlStream#11318800
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Sending command [CWD //public_html]
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Received response [250 OK. Current directory is /public_html]
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Sending command [PASV]
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Received response [227 Entering Passive Mode (192,64,117,187,47,58)]
System.Net Information: 0 : [12064] FtpControlStream#11318800 - Sending command [LIST ctc]
System.Net Information: 0 : [12064] FtpWebRequest#41130254::(Releasing FTP connection#11318800.)
System.Net Error: 0 : [12064] Exception in FtpWebRequest#41130254::GetResponse - The underlying connection was closed: An unexpected error occurred on a receive..
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.GetResponse()
System.Net Information: 0 : [12064] FtpControlStream#33675143 - Received response [226-Options: -a 
226 133 matches total]
System.Net Information: 0 : [12064] FtpWebRequest#21454193::(Releasing FTP connection#33675143.)

So any idea what am I doing wrong?