Construct a multipart MIME message without holding it in memory

Question

I want to create a multipart MIME message out of some part data that I have available from some streams (eg open file handle) and I want the resultant multipart MIME message to be sent out through a stream (eg network port as part of HTTP connection).

However, these parts are numerous GB in size so I don't want to have to hold them in memory, and I don't want the multipart MIME message to ever need to be fully in memory.

So how can I create multipart MIME message without holding it in memory, ie streaming in the parts and having it streaming out the constructed message 'on the fly'?

I'd like to use a 3rd party MIME library but don't know if any support streaming like this, and I'm reluctant to author my own.

Any recommended practice?

Answer 1

After much further studying (inspired by MimeKit example above) that the .Net HTTP classes support what I was looking for in a similar way to how MimeKit does.

Create a MultipartContent and add parts using StreamContent, then get a stream for the multipart message using MultipartContent.ReadAsStreamAsync and the part streams are not read until the multipart streaming needs it. This means (from monitoring process size) the original parts are never completely in memory at any time, nor is the multipart message.

Answer 2

MimeKit supports streaming the content of MimeParts from disk rather than from memory, but I'm not sure if any HTTP libraries will provide you the raw Socket or NetworkStream (or SslStream) to write the message to.

Assuming the HTTP library is not a limiting factor, you can create MimeParts in MimeKit like this:

var part = new MimePart ("application", "octet-stream") {
    Content = new MimeContent (File.OpenRead ("large-file.dat")),
    ContentTransferEncoding = ContentENcoding.Base64,
    FileName = "large-file.dat"
};

You can create a multipart like this:

var multipart = new Multipart ("mixed");
multipart.Add (part);

You can set the multipart as the body of a MimeMessage like this (although I'm not sure you really need the MimeMessage container for what you intend to do):

message.Body = multipart;

And you can stream the message (or the multipart) to a network stream (or any type of stream) like this:

message.WriteTo (stream);

If you need to make sure that the message is converted to use CRLF even on Unix systems, you can do this:

var options = FormatOptions.Default.Clone ();
options.NewLineFormat = NewLineFormat.Dos;

message.WriteTo (options, stream);

This will push the message data through a filter that will convert line endings (if they need to be converted) as they are being written to the output stream.

Since message data is chunked in approximately 4k blocks, it should fit your requirements.

If you have further questions, find my email address on GitHub (same alias and I'm the author of MimeKit/MailKit, so I should be easy to find) and shoot me an email. I'll be happy to answer any questions you have about how to do this.

In fact, MimeKit was designed to do exactly what you want to do and to make sure it all worked, I wrote MailKit's original SmtpClient. Then I wrote the Pop3Client the weekend afterward to test that I could parse a message as it was read directly from a socket.

So yea, it can do what you need it to do.