OpenSSL read client certificate error using libev with non-blocking sockets

Question

I've spent some time searching the interwebs looking for a better way to analyze and debug my issue, but I can't seem to find a solution. So I figured I'd ask.

Briefly. I'm attempting to create a non-blocking ssl forwarding proxy. The server portion of the proxy is using a self signed server certificate, which I signed using my own CA certificate. If it matters, I'm using libev. I successfully created a non-encrypted proxy first (it blindly forwarded web traffic), and now I'm trying to add SSL to it. :)

I'm having issues getting the client to connect to the proxy. I've tried both wget and ssl's s_client as test clients, as I wanted to have some automated testing.

ssl server setup (this code is called from the libev watcher listening socket accept_handler(), on a EV_READ event):

/* setup client side ssl state (we are a SERVER) */
ctx = SSL_CTX_new(SSLv23_server_mode());
SSL_CTX_set_options(ctx, SSL_OP_NO_SSLV2 | SSL_OP_ALL);
SSL_CTX_set_info_callback(ctx, client_info_cb);
SSL_CTX_set_cipher_list(ctx, "ALL:!SSLv2:-aNULL");
//SSL_CTX_load_verify_locations(ctx, CA_CERTIFICATE, NULL);
//SSL_CTX_set_client_CA_list(ctx, SSL_load_client_CA_file(CA_CERTIFICATE));
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);
SSL_CTX_set_verify_depth(ctx, 0);
SSL *client_ssl = SSN_new(ctx);
SSL_set_mode(client_ssl, SSL_MODE_ENABLE_PARTIAL_WRITE);
SSL_set_accept_state(client_ssl);
SSL_set_fd(client_ssl, client_fd);

/* initialize client handshake watchers */
ev_io_init(&ev_r_ch, client_handshake, client_fd, EV_READ);
ev_io_init(&ev_w_ch, client_handshake, client_fd, EV_WRITE);
... other watcher inits and set watcher data portions ...

/* start the read */
ev_io_start(loop, &ev_r_ch);

The libev loop was setup as:

loop = ev_default_loop(EVFLAG_AUTO);

I've got timers and such to check for a shutdown flag as well as other housekeeping activities.

My client_handshake() main function looks like this essentially:

int t = SSL_accept(client_ssl);
if (t == 1) { // SSL_ERROR_NONE
    end_client_handshake(...);
} else {
    int err = SSL_get_error(client_ssl, t);
    if (err == SSL_ERROR_WANT_READ) {
        ev_io_stop(loop, &ev_w_ch);
        ev_io_start(loop, &ev_r_ch);
    }
    else if (err == SSL_ERROR_WANT_WRITE) {
        ev_io_stop(loop, &ev_r_ch);
        ev_io_start(loop, &ev_w_ch);
    }
    else ...
}

In the client_info_cb() I print out the internal SSL state as we progress, and get the following from my print() function:

client_info_cb: 8193: SSLv3 read client hello A
client_info_cb: 8193: SSLv3 write server hello A
client_info_cb: 8193: SSLv3 write certificate A
client_info_cb: 8193: SSLv3 write server done A
client_info_cb: 8193: SSLv3 flush data
client_info_cb: 8194: SSLv3 read client certificate A
client_info_cb: 8194: SSLv3 read client certificate A

And this is where it hangs. I tried modifying the client_handshake() function to loop(1) {} around SSL_accept() if I detected the SSL_ERROR_WANT_READ (which is what SSL_get_error() returns after the second "read client certificate A" message above).

That did nothing but put me into an infinite loop(), continuously calling SSL_accept().

I'm assuming that the SSL state machine needs some additional information that it cannot get. I at first thought that I needed to continue reading from the socket, but that didn't work.

Also, I'm confused as to why my proxy is trying to read a client certificate, as I've explicitly specified that I don't want to verify the client certificate (SSL_VERIFY_NONE) above; unless I'm misunderstanding the purpose of that function.

If anyone has any insight on this, I'd be grateful. Or perhaps a better method of debugging this issue. strace() is useless for this, and I don't get any good return/error messages out of either wget or s_client.

I tried setting up alert_callbacks and msg_callbacks within SSL's state machine, but that didn't give me any more information than the info callback did.

At this point I'm not sure if it's a socket problem, or a SSL problem, or what.

edit1: I'd like to point out that in the accept_handler(), I'm first connecting to the server over ssl, in order to validate the certificate of the host I'm proxying, before finishing the accept(). If I reverse the order of operations, and accept() first before connecting() onward, it works.

edit2: I tried looking at the tcpdump output between s_client and the proxy. After the write server data and flush data referenced in the client_info_cb, the client sends a "Client Key Exchange", "Change Cipher Spec", and "Encrypted Finished Message". However the ssl state machine is looking for a client certificate???

--> Client Key Exchange
write to 0x9547a78 [0x9592e90] (523 bytes => 523 (0x20B))
0000 - 16 03 01 02 06 10 00 02-02 02 00 be 51 c7 3d 77   ............Q.=w
0010 - 5a b3 9e 28 81 f4 4e b5-63 ce ce 0b 19 f3 85 64   Z..(..N.c......d
0020 - 29 0e e8 22 83 b8 60 a6-54 e3 7a 62 b3 37 d8 04   ).."..`.T.zb.7..
0030 - 6c f1 8e ff 50 44 ed cc-7b 08 61 0c 16 88 f4 61   l...PD..{.a....a
0040 - 7b 8d f2 1e 04 1d 74 3d-cc ee a4 93 d3 bb 90 ee   {.....t=........
<snip>
--> Change Cipher Spec
write to 0x9547a78 [0x9592e90] (6 bytes => 6 (0x6))
0000 - 14 03 01 00 01 01
--> Finished Message                                 ......
write to 0x9547a78 [0x9592e90] (53 bytes => 53 (0x35))
0000 - 16 03 01 00 30 9a 88 8b-14 d6 d1 f1 f7 d8 0d ac   ....0...........
0010 - 38 cd 54 78 26 85 7b 11-c8 e9 db 8d a2 0c 6a a8   8.Tx&.{.......j.
0020 - d4 e7 d4 ad 5d 7a 6d 47-eb f9 5f 2c f6 ca 6a 1f   ....]zmG.._,..j.
0030 - 17 a6 58 25 41                                    ..X%A

Answer 1

Maybe you could be experiencing some session cache problems (it's enabled by default and "edit 1" points to it), ¿are you trying to serve to the client an spoofed cert from the server? Maybe the internal cache is stored by cert name, and it tries to resume the server session...

Try switching off the cache with SSL_CTX_set_session_cache_mode with the SSL_SESS_CACHE_OFF option.

About the "edit 2" the client key exchange phase do not talks about sending the client's certificate, it sends a secret (or nothing) used for the selected cipher system (it can send an empty client exchange if the selected cipher does not need a secret). Read here about it.

Answer 2

Did you set the client_fd socket to a non-blocking mode in a first place?

int fd_nonblock(int client_fd)
{
    int flags = fcntl(client_fd, F_GETFL, 0);
    flags |= O_NONBLOCK;
    return fcntl(client_fd, F_SETFL, flags);
}

Answer 3

Just use OpenSSL bufferevents from libevent API instead. You'll get the same or even better functionality. I'm not sure but it seems like you are trying to implement SSL server at very low level, but, for instance, it would be painfull with some native Linux asynchronous I/O mechanisms.