openssl 1.0.2, how to force server to choose only set of ciphers

Question

I have client server which uses opensl 1.0.2j, and want to force the server to use only the following ciphers.

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDH-RSA-AES128-GCM-SHA256
ECDH-RSA-AES128-SHA256
ECDH-ECDSA-AES128-GCM-SHA256
ECDH-ECDSA-AES128-SHA256
DHE-DSS-AES256-GCM-SHA384
DHE-DSS-AES256-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-SHA256
DHE-DSS-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-SHA256
DHE-DSS-AES128-SHA256

My server side code will look like below.

method = SSLv23_server_method();
ctx = SSL_CTX_new(method);
SSL_CTX_set_cipher_list(ctx, "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDH-RSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:DHE-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256");
SSL_CTX_set_ecdh_auto(ctx, 1);
SSL_CTX_use_certificate_file(ctx, certFilePath, SSL_FILETYPE_PEM);
SSL_CTX_use_PrivateKey_file(ctx, privKeyPath, SSL_FILETYPE_PEM)
SSL_accept()

The last step ssl_accept fails with

here'error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher'

I have error checking for each ssl calls in the above code and for clarity purpose not put all the code. If I use "TLSv1.2:!ADH:!NULL" for SSL_CTX_set_cipher_list() it works fine.

Edit: The key generated is RSA:4096. Do I need to generate keys differently for ECDH/ECDHE/DHE?

Could you please help me to find why it fails and how can I resolve it?

Let me know if you need more information.

Thanks, Naga

Answer 1

Based on this code you don't set any DH parameter so any of these DHE-* certificates will not work. Also, no static parameter for ECDH (not ECDHE) are set so no ECDH-* ciphers will be used either. This leaves only:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA384

But from these 4 ciphers only two can be used since either you have a RSA certificate (first two ciphers) or an ECC certificate (last two ciphers). Most likely it is an RSA certificate which leaves:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA384

Since you get no shared cipher it is likely that your unknown client does not support any of these two ciphers.

If I use "TLSv1.2:!ADH:!NULL" for SSL_CTX_set_cipher_list() it works fine.

With OpenSSL 1.0.2 in a common configuration I see that this set also includes the following ciphers (skipping all DH, ECDH-... as before):

AES256-GCM-SHA384
AES256-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA256
AES128-GCM-SHA256
AES128-SHA256

So it is likely that your unknown client use any of these ciphers to connect successfully. For more on this you need to look at the specific client and maybe its configuration.