Interoperability problems python2 python3

Question

Two uServices are communicating via a message queue (RabbitMQ). The data is encoded using message pack.

I have the following scenarios:

• python3 -> python3: working fine
• python2 -> python3: encoding issues

Encoding is done with:

umsgpack.packb(data)

Decoding with:

umsgpack.unpackb(body)

When doing encoding and decoding in python3 I get:

data={'sender': 'producer-big-red-tiger', 'json': '{"msg": "hi"}', 'servicename': 'echo', 'command': 'run'}

When doing encoding in python2 and decoding on python3 I get:

data={b'command': b'run', b'json': b'{"msg": ""}', b'servicename': b'echo', b'sender': b'bla-blah'}

Why is the data non "completely" decoded? What should I do on the sender / receiver to achieve compatibility between python2 and python3?

Answer 1

Look at the "Notes" section of the README from msgpack-python;

msgpack can distinguish string and binary type for now. But it is not like Python 2. Python 2 added unicode string. But msgpack renamed raw to str and added bin type. It is because keep compatibility with data created by old libs. raw was used for text more than binary.

Currently, while msgpack-python supports new bin type, default setting doesn't use it and decodes raw as bytes instead of unicode (str in Python 3).

You can change this by using use_bin_type=True option in Packer and encoding="utf-8" option in Unpacker.

>>> import msgpack
>>> packed = msgpack.packb([b'spam', u'egg'], use_bin_type=True)
>>> msgpack.unpackb(packed, encoding='utf-8')
['spam', u'egg']