How do I cleanly shutdown zeromq DEALER/ROUTER inproc connections

Question

In one of my applications I'm using DEALER/ROUTER inproc connections. I set the linger option on the DEALER socket to 0, so that all messages sent on the DEALER socket shall be discarded once the ROUTER socket is closed. Although this works well for TCP connections, it blocks for inproc. Here's a minimal working example:

#include <zmq.h>

#include <windows.h>

int main()
{
    void *context = zmq_ctx_new();

    void *router = zmq_socket(context, ZMQ_ROUTER);
    zmq_bind(router, "inproc://socket");

    void *dealer = zmq_socket(context, ZMQ_DEALER);
    zmq_connect(dealer, "inproc://socket");

    int linger = 0;
    zmq_setsockopt(dealer, ZMQ_LINGER, &linger, sizeof(linger));

    zmq_close(router);

    // sleep for 1 ms
    Sleep(1);

    // this call blocks
    zmq_send(dealer, "message", 7, 0);

    zmq_close(dealer);
    zmq_ctx_destroy(context);

    return 0;
}

Before the DEALER socket can be closed, the zmq_send() call blocks. In this minimal example, I had to add a Sleep(1) call. When this call is omitted, zmq_send() doesn't block. When blocked, the call stack is as follows:

[External Code] 
libzmq.dll!zmq::signaler_t::wait(int timeout_) Line 253 C++
libzmq.dll!zmq::mailbox_t::recv(zmq::command_t * cmd_, int timeout_) Line 80    C++
libzmq.dll!zmq::socket_base_t::process_commands(int timeout_, bool throttle_) Line 1023 C++
libzmq.dll!zmq::socket_base_t::send(zmq::msg_t * msg_, int flags_) Line 869 C++
libzmq.dll!s_sendmsg(zmq::socket_base_t * s_, zmq_msg_t * msg_, int flags_) Line 346    C++
libzmq.dll!zmq_send(void * s_, const void * buf_, unsigned __int64 len_, int flags_) Line 371   C++

I'm using Windows 10 x64, libzmq 4.2.1 (tested it with 4.1.6 as well), and Visual Studio 2015. How can I cleanly shut down the DEALER/ROUTER connection? Is this a bug in libzmq?

Answer 1

Would it be feasable to use ZMQ_DONTWAIT in your

zmq_send()

call and evaluate the errorcode?

Changing your code to

// this call blocks no more
zmq_send(dealer, "message", 7, ZMQ_DONTWAIT);
int ec = zmq_errno();
printf("code: %d\nstring: %s\n", ec, zmq_strerror(ec));

would result in

code: 11

string: Resource temporarily unavailable

Alternatively, can a socket monitor be used to detect the close event and prevent further sending? This does depend on your architecture.

Answer 2

inproc:// is not a true transport protocol. because of this ZMQ_LINGER has no effect here. additionally, inproc must have a socket connected for a receiving end. non inproc transport types have various states that inproc doesn't share, and various sockets will behave differently when not in a connected state. req will block until a connection becomes available. dealer however will not normally block, but will queue the message until a connection becomes available. inproc however does not have independent IO threads or independent IO queues as is defined in zmtp. as a result, if there is no reciving end it will block, even if a non-blocking type. whats happening in your code is that the router is not fully closed by the time you try sending on dealer, so inproc is still in a state where it thinks it can send even though it will have no possible endpoint when the zmq_close operation completes. waiting the additional time allows the operation to complete, and therefore inproc is in a state that it can't queue and will therefore block. chances are, without the sleep, the message will be lost or will cause some other unexpected behavior. this is one if the reasons inoroc started out as just ZMQ_PAIR.