ArgumentException {token} while deserializing using protobuf.net

Question

I was using protobuf-net for serialization and deserialization.

The method i was following is creating a RuntimeTypeModel and adding all the types that needs to be serialized or to be supported for seralization. Upon adding all the types, i am using the TypeModel.Compile() for performance benefits.

When a class instance is serialized and saved as blob into the database, i was able to deserialize that when fetching from database. However, when i change the machine and try to deserialize, I am getting ArgumentException of type token at

   at ProtoBuf.ProtoReader.EndSubItem(SubItemToken token, ProtoReader reader) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 584
   at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 567
   at ProtoBuf.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 543
   at proto_157(Object , ProtoReader )
   at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line 57
   at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source) in c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 783
   at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556
   at ProtoBuf.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 543
   at proto_190(Object , ProtoReader )
   at ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line 57
   at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source) in c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 783
   at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 683
   at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 582
   at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 561

The codebase is same, but run from a different machine i was encountering this problem.

The classes that participate in serialization is pretty simple

public class NetworkData
{

}

/// <summary>
/// Class for representing dashboard URL item
/// </summary>
public class DashboardURLItem : NetworkData
{
    /// <summary>
    /// Gets and Sets the URL
    /// </summary>
    public string URL { get; set; }

    /// <summary>
    /// Gets and Sets the Header for the URL Item
    /// </summary>
    public string Header { get; set; }
}

For serialization i am using the following code

    public byte[] SerializeData(T piclObjectData)
    {
        try
        {
            using (MemoryStream loclStream = new MemoryStream())
            {
                NWTypeModel.Serialize(loclStream, piclObjectData);
                byte[] resultbuffer = new byte[loclStream.Length + 4]; // Those many bytes were generated in serialization + Message Length
                Array.Copy(BitConverter.GetBytes(resultbuffer.Length), resultbuffer, 4);
                Array.Copy(loclStream.GetBuffer(), 0, resultbuffer, 4, loclStream.Position);
                return resultbuffer;
            }
        }
        catch (Exception E)
        {
            Logger.WriteLog(ToString() + ": Exception while serializing - " + E.Message);
            throw;
        }
    }

Code for deserialization

    public T DeSerializeData(byte[] piarBuffer)
    {
        try
        {
            using (MemoryStream loclStream = new MemoryStream())
            {
                loclStream.Seek(0, SeekOrigin.Begin);
                loclStream.Write(piarBuffer, 4, piarBuffer.Length - 4); // 4 Bytes for Message length
                loclStream.Seek(0, SeekOrigin.Begin);
                return (T)NWTypeModel.Deserialize(loclStream, null, f_GenericType);
            }
        }
        catch (Exception E)
        {
            Logger.WriteLog(ToString() + ": Exception while de-serializing - " + E.Message);
        }
        return null;
    }

Serialization and deserialization was working fine on same machine for the database, but when getting the value from the database on another machine, it is not able to deserialize.

Please throw some light if i am doing anything wrong or need to take care of anything more.

Thanks

Answer 1

Finally after spending hours of time debugging the protobuf-net source code, i found out the problem. Protobuf-net internally uses a list and there is no sorting on the types that will be added. As such, when i start the exe from Machine A, the types using reflection was getting added in a different order compared to the ordering on Machine B. Don't know if reflection should emit same order of types when the code base is same. However, i have added a SortedList to store the types in an assembly, then adding to RuntimeTypeModel.

Now, since both Machine A and Machine B has the same code to sort the types and there will be no extra types as the code is same, i have ended up with proper ordering of types on both sides and got the proper deserialized object. Machine A serializes an object and stores in a blob field in Table, Machine B reads from the table, DeSerializes and then consumes the object.

The ordering is important because, i am saving the serialized data in database and then deserializing. but, if in future more types get added, then the order will get changed again. I am trying for a fool proof solution to this so that future types can be added without breaking the serialization or deserialization of existing types.

There should be a method in which the type's order should be preserved and if any new type is detected, then that should be appended at the last, so that existing types MetaTypes are not disturbed.