RST after Client hello

Question

Boost beast TLS client (largely based on this) which is not connecting to Microsoft Azure azurewebsites hosted web app. Viewing the Wireshark output, it appears that the server is sending RST after the client sends the Client hello message, which details the available ciphers. The client works with other websocket servers, running Java Jetty and a let's encrypt certificate.

The preferred cipher, which firefox appears to use when connecting to the domain, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, appears by default, in both the cipher list on the host and in the packet sent by the client in the Client hello message.

I've tried other things such as disabling the client side certificate authentication, which has fixed other certificate issues in the past. I've been able to modify the cipher suites, using the openssl library and the native_handle. But this hasn't fix the issue, I found one stack overflow question, where the answer suggests the cipher has compatibility issues.

Googling RST message after Client hello, appears to return issues mainly with IIS and Windows servers.

Does anyone have any idea what this could be?

Answer 1

I think ciphers are not only cause of RST packet. As you have already validated ciphers are present at both places, perhaps I strongly suggest to confirm the signature algorithm then it could be the certificate binding type you should consider to check.

I have seen JAVA based server throws RST packet if certificate is using SNI binding.

Give it a try and check certificate binding and confirmed otherwise, suggest to update the client as server is azure websites which is PAAS offering and no scope for any changes inside azure websites deployment servers.

you can also use openssl command with -showcerts switch to compare the certificate and their ciphers and signature algorithm

Answer 2

So I just happened to try the synchronous example out, which connected. One of the small number of differences between that and our code, in the client hello packet, was the SNI / server name string; Server Name: xxx.azurewebsites.net:443 vs Server Name: xxx.azurewebsites.net.

It happened to include the port in our embedded code, but not from that example. Our code happens to be based off the async client example.

The async example has;

    // Update the host_ string. This will provide the value of the
    // Host HTTP header during the WebSocket handshake.
    // See https://tools.ietf.org/html/rfc7230#section-5.4
    host_ += ':' + std::to_string(ep.port());

    // Set a timeout on the operation
    beast::get_lowest_layer(ws_).expires_after(std::chrono::seconds(30));

    // Set SNI Hostname (many hosts need this to handshake successfully)
    if(! SSL_set_tlsext_host_name(
            ws_.next_layer().native_handle(),
            host_.c_str()))
    {
        ec = beast::error_code(static_cast<int>(::ERR_get_error()),
            net::error::get_ssl_category());
        return fail(ec, "connect");
    }

The sync example has;

    // Set SNI Hostname (many hosts need this to handshake successfully)
    if(! SSL_set_tlsext_host_name(ws.next_layer().native_handle(), host.c_str()))
        throw beast::system_error(
            beast::error_code(
                static_cast<int>(::ERR_get_error()),
                net::error::get_ssl_category()),
            "Failed to set SNI Hostname");

    // Update the host_ string. This will provide the value of the
    // Host HTTP header during the WebSocket handshake.
    // See https://tools.ietf.org/html/rfc7230#section-5.4
    host += ':' + std::to_string(ep.port());

So basically, the examples work differently, either adding the port to the SNI or not. Going on the github for boost beast, shows that the async example has been fixed in develop; and they've chosen not to include the port number.