I have beeen using ZMQ (via pyzmq) recently for setting up publish/subscribe between nodes on a single machine. However, when things go wrong, I have no visibility into what is causing the issue, and it's causing me a great deal of grief.
Right now, I am publishing a message and then attempting to receive it on a connected subscriber socket. Or, at least, I think it's a connected subscriber socket. I don't really know, because as far as I know, ZMQ does not provide any way to check this! When I try to create a new subscriber socket within the same context, that command hangs indefinitely. No error is thrown (like it would be if the context had been terminated), and it doesn't run in the background like it would if the publish socket cannot be found. Again, I have no visibility into what is going on and what the problem is.
I'm looking for suggestions on generally getting visibility into what ZMQ is doing and how to debug it when things aren't going as expected. Obviously, I can check whether a TCP port on my local machine has been opened to verify that a "publish" socket has been created. But how do I know when a subscribe socket is connected to that "publish" one? How can I tell - indirectly or directly - what is going on?
You can use zmq_socket_monitor() to get a connection from inside the socket's state machine, allowing you to see what is going on. I'm pretty sure the python implementations have this too.
Further ideas include using WireShark. This understands ZMQ's zmtp, and can disect it. So you can see on the network what is going on between the connected machines. This allows you to see the connection / message transfer processes happening, and also the content of messages you're sending. If those messages are encoded somehow (e.g. Google Protocol Buffers), you can build in a custom disector for your particular GPB schema.
Between these two, you can get a very thorough view within your application, and on the wire.