Big files deserializations by protobuf-net

Question

I need to deserialize 1.5GB txt file. I am using protobuf-net from code.google.com/p/protobuf-net/

Sometimes it fails (about 50% cases) with different exceptions (null reference, memory access violation) in different places. I have noticed that if processor is lowly loaded then probability of failure is decreasing.

What should I do to avoid such failures?

Here is the example of deserializing code:

public static History LoadFromFile(string path)
    {
        using (var fileStream = File.OpenRead(path))
        {
            var obj = Serializer.Deserialize<History>(fileStream);                
            return obj;
        }
    }

Today I have a FatalExecutionEngineError with error code 0xc0000005, but I can't realize what part of code is possibly unsafe. It's not a constant error, everything works correctly after I restart the application.

Here is the example of files with serialization, which I need to deserialize: https://docs.google.com/file/d/0B1XaGInC6jg3ZXBZZDA3bHh3bVk/edit

Answer 1

Google :

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy. That said, Protocol Buffers are great for handling individual messages within a large data set.

Source link