SerializeAsString - server, ParseFromString - client. "Decode error" Exception occurred, but the data looks correct

Question

Our application workflow uses protobuf for cross-language IPC. We have a C++ application publishing data over Linux shared memory to various clients on the same host. Data is published to shared memory using the protobuf API of "SerializeAsString", and the client's side does "ParseFromString". Some of the clients are written in Python, while others are written in C++. Even though the data we get after parsing appears to be fine, in C++ the "ParseFromString" method always returns false. In the beginning we used protobuf v3.15.5 on the Python clients, we got "RuntimeWarning: Unexpected end-group tag: Not all data was converted", from ParseFromString() call.

After upgrading server and client side protobuf to 21.12, we start getting Decode error exception. google.protobuf.message.DecodeError: Error parsing message Again, the strange thing is that all the data looks fine despite the exceptions. Any suggestions?

Language: C++//Python

operating system :

• Server - Docker image Ubuntu 20.04.5 run on aarch64.

• Client - Docker image Ubuntu 20.04.5 run on x86.

Runtime / compiler are you using - Python 3.10, Gcc/G++ - 9.

What did you do? Steps to reproduce the behavior:

Part of my Proto:

message Frame
{
  bytes depth           = 1;
  bytes rgb             = 2;
  uint64 SampleTime     = 3;
  uint64 SentTime       = 4;
  uint64 AlignTime      = 5;
}
message CameraData
{
  Frame                 frame           = 1;
  uint32                fps             = 2;
}

Serialize with: SerializeAsString()

data blob : b'\rQ\xc3\x8e\xb6\x15\xd6\xae\xc7\xb8\x1d\x01m-\xb7%\xe6\xb2@?-\x12\xef\x11?5\xec\xf00>=\xdb\x85\x8f>E;\xd3j\xb9M\xa49\xa4\xbbUj\xa1\x0e\xba]\xfcL1\xbaeI\x03\x04\xbbmD\x98\xf7\xbau\xa3\x10\xc2\xbb}*\xb8\x07\xbe\x85\x01\x0f\xbfk\xbc\x8d\x01\x8f\xcc\xe7>\x95\x01\x92I0\xbf\x9d\x01\xee\xa2\xef\xbf\xa0\x01\x02\xad\x01\xff\xff\x7f\x7f\xb5\x01\xff\xff\x7f\x7f\xc0\x01\xd5\xc4\x9b\xaf\x91\xf4\xfc\x02\xc8\x01\xdb\xf2\x9b\xaf\x91\xf4\xfc\x02\xd2\x01\x02\x08\x01'

Parse with : ParseFromString()

error: google.protobuf.message.DecodeError: Error parsing message

Thanks a lot for your Help !

Answer 1

I am working with Itzik, you are right, the schema is not the same as the serialized data. this is the correct schema:

message ExposureState { bool AutoExposure = 1; uint32 ExposureValue = 2;}
message TData {
float X = 1;
float Y = 2;
float Z = 3;
float qX = 4;
float qY = 5;
float qZ = 6;
float qW = 7;
float X_velocity = 8;
float Y_velocity = 9;
float Z_velocity = 10;
float X_angular_velocity = 11;
float Y_angular_velocity = 12;
float Z_angular_velocity = 13;
float X_acceleration = 14;
float Y_acceleration = 15;
float Z_acceleration = 16;
float X_angular_acceleration = 17;
float Y_angular_acceleration = 18;
float Z_angular_acceleration = 19;
uint32 confidence = 20;
float asic_temperature = 21;
float motion_module_temperature = 22;
uint32 fps = 23;
uint64 SampleTime = 24;
uint64 SentTime = 25;
ExposureState Exposure = 26; }