How do I correctly enforce OpenSSL certificates?

Question

I'm currently tinkering on what is effectively a chat server. Since I do not want to expose my users too much, I added TLS encryption to it using LibreSSL's fork of the OpenSSL library. The rest of the code appears to work fine, but I think I am not doing the certificates correctly.

I have a private/public certificate on the server which should be used not only to encrypt the communication, but also to ensure the server is really who the client wanted to talk to. And that's the part I can't figure out:

1. How do I give the server's public key to the client? It needs it to verify that it's talking to the right server. Or should I be doing something else, maybe involving the root CA's certificate? Is there API to provide that? I can package the public key with the executable as a .pem, but I can't find the API to tell OpenSSL about a public server key to use for client requests, or the root CA.

2. How do I get the system's certificate for the client? Right now I just created one in a .pem file, but I don't really want to have to build a new download with a unique certificate for every user downloading the client. Surely there's a way to get "the current user's certificate" or auto-generate one for this use via some OpenSSL API?

If anyone could point me at the right API to use, that'd be great! I'd also take clues, links to similar questions on SO, tutorials, pointers at books aimed at OpenSSL-crypto-beginners, answers or sample code.

Currently, I'm using both SSL_CTX_use_certificate_file() and SSL_CTX_use_PrivateKey_file() to set certificates in both the client and the server program. You can see the code in the above linked Github chat server repository, in eleven_session.cpp

Answer 1

1: So, fist of all, you don't need to give the server's public key to the client, because the OpenSSL API will do it for you. You create a socket using the regular socket function and then transfer it to OpenSSL, and this socket is the identification for every client that connects to the server. Then you use this SSL Socket for example in the socket write function SSL_write to send encrypted text to the client.

2: When I started i started using encrypted sockets, I also had a lot of problems finding the right way to create a working .pemfile, so here's my try how I've done it. You have to use linux terminal and you have to install OpenSSL using sudo apt-get install openssl:

openssl genrsa -des3 -out self-ssl.key 2048
openssl req -new -key self-ssl.key -out self-ssl.csr
cp -v self-ssl.{key,original}
openssl rsa -in self-ssl.original -out self-ssl.key
rm -v self-ssl.original
openssl x509 -req -days 3650 -in self-ssl.csr -signkey self-ssl.key -out self-ssl.crt
cat self-ssl.crt self-ssl.key > server.pem

This file is then used like you've already found out with these 2 functions: SSL_CTX_use_certificate_file and SSL_CTX_use_PrivateKey_file

Try it with that .pemfile, if it still doesn't work, write it into comments.

Hope this will help you.

Answer 2

Thanks to SteffenUllrich and schacker22 for helping me with this issue. Here's the solution I cobbled together with their guidance:

After connecting, the client uses BIO_new(BIO_s_file()), BIO_read_filename() and PEM_read_bio_X509_AUX() to load the public certificate of the server from a PEM file, then compares it to the certificate returned by SSL_get_peer_certificate() using X509_cmp(). If that doesn't give 0, the client aborts the connection because it is not talking to the correct server.

Only the server gets a certificate, which I set using SSL_CTX_use_certificate_file() (public) and SSL_CTX_use_PrivateKey_file() to point to the same PEM file containing both the public and private key.

I decided not to use the verify callback, as it would replace the existing certificate verification, so I'd have to do that part as well. The load_cert() function in LibreSSL's apps.c file was a great help in figuring out how to load the certificate.

If you're interested in seeing the final source code, it's publicly available from my Git repository for the eleven chat server and client.

This is my first time working with SSL sockets, so if you spot anything wrong here, please let me know. The whole point of this exercise is to learn how to do it right.