Azure: How to reliable download a file from blobstorage?

Question

We create a lot of csv files and store them in an azure blob container. When all files have been created, they shall be downloaded to a certain location within a vpn. This works fine most of the time, but it can happen that some files are much smaller than expected.

The files have an average size of ~40 mb, sometimes the files are between 7 and 30 mb. This is the code so far:

private void DownloadBlobToFile(ICloudBlob blob, string fileName)
{
    _log.Debug("DownloadBlobToFile");
    Contract.Requires<ArgumentNullException>(blob == null, "blob must not be null.");
    Contract.Requires<ArgumentNullException>(string.IsNullOrWhiteSpace(fileName), "fileName must not be null or empty.");

    using (var fileStream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read))
    {
        blob.DownloadToStream(fileStream, null, GetBlobRequestOptions());
        _log.Info(string.Format("File {0} successfully downloaded.", fileStream.Name));
    }
}

private BlobRequestOptions GetBlobRequestOptions()
{
    if (_blobRequestOptions != null)
    {
        return _blobRequestOptions;
    }
    _blobRequestOptions = new BlobRequestOptions
    {
        ServerTimeout = TimeSpan.FromSeconds(60),
        MaximumExecutionTime = TimeSpan.FromSeconds(180),
        RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(3), 5)
    };
    return _blobRequestOptions;
}

When the files are downloaded incompletely, there is no error message at all. What is the best procedure to check if files have been successfully and completely downloaded?

Answer 1

Not sure if you have control over the name of the file, but if you do you can name the file with the MD5 hash of the file content. Then you can compute that MD5 hash when the file downloads to make sure it matches the file name. This way you would know if you have all of the content. You could also store this metadata somewhere else and associate it, but the key is using a MD5 hash to validate you have everything you expect to have.