protobuf-net Stream is disposed before returning data

Question

I have a webapi that returns some protobuf-net serialized items after being transformed from a database column. The number of items can be rather large so I'd like to avoid materializing them all in memory and rather stream them out however, I never get a single one out before my Stream throws an already disposed exception. If I materialize the list it does work but I'd like to avoid it. Here is what I'm doing::

private override async Task<IEnumerable<MyObj>> GetRawData(int id)
{
    using(var sqlConn = GetOpenConnection())
    {
        using (var sqlReader =(await sqlConn.ExecuteReaderAsync(
            _workingQuery,new {id = id}, 60)
                    .ConfigureAwait(false) as SqlDataReader))
        {
            await sqlReader.ReadAsync().ConfigureAwait(false);

            using (var stream = sqlReader.GetStream(0))
            {
               return _serializerModel.DeserializeItems<MyObj>(stream, PrefixStyle.Base128, 1)
            }                
        }
    }
}

private async Task<IHttpActionResult> TransformData(int id)
{   
    var data=await GetRawData().ConfigureAwait(false);
    return Ok(new {Data=data.Where(m=>...).Select(m=>...)})
}
[HttpGet,Route("api/myroute/{id}"]
public async Task<IHttpActionResult> GetObjs(int id)
{
     return await TransformData();
}

However, I end getting an error about reading a disposed stream. How can I avoid this error?

Answer 1

The long and short of it is that you are returning a non-enumerated sequence, and closing (disposing) everything it needs before it ever gets to the caller. You're going to need to either:

• enumerate the sequence eagerly (buffer in memory) - for example, adding .ToList()
• restructure the code so that nothing is disposed until the end of the iteration

For the latter, I would be tempted to use an iterator block for the central part (after all the await). Something like:

bool dispose = true;
SqlConnection conn = null;
//...others
try {
   conn = ...
   ...await...
   var result = InnerMethod(conn, stream, ..., ) 
    // ^^^ everything now known / obtained from DB
    dispose = false; // handing lifetime management to the inner method
    return result;
} finally {
    if(dispose) {
        using(conn) {}
        //.//
    }
}

IEnumerable<...> Inner(...) { // not async
    using (conn)
    using (stream) {
         foreach(var item in ...) {
             yield return item;
         }
    }
}

Answer 2

This line

return _serializerModel.DeserializeItems<MyObj>(stream, PrefixStyle.Base128, 1)

Returns just an indexer. As you seem to be aware the result is not yet materialized. However, right after the return, the using block disposes your Stream.

The solution would be to have GetRawData return the Stream. Then, inside TransformData, inside a using block for the Stream, you deserialize, filter and return the results.

The second potential issue is that your current approach processes the whole result and then sends it to the client all at once.

In order to have Web API send a streamed result you need to use HttpResponseMessage, and pass a Stream to it's Content property.

Here's an example: http://weblogs.asp.net/andresv/asynchronous-streaming-in-asp-net-webapi

When you want to stream a Stream to another Stream (I know, it sounds funny), you need to keep the ReaderStream open until the WriterStream is done writing. This is a high level representation:

using (ReaderStream)
{
    using (WriterStream)
    {
        // write object from ReaderStream to WriterStream
    }
}