The main problem is that I'm quite new to C, and the OpenSSL documentation is not clear enough for me, I've tried using Reading and writing rsa keys to a pem file in C, but I don't quite understand it. For example, how does the create_rsa_key() function creates both the private and the public? And where does pCipher come from? And what's the point of the pcszPassphrase?
I'll explain as if it was some sort of pseudocode, this is what I want to do, and the parts in bold are those I don't know how to do it:
- generate private key and public key as buffers in hex (client)
- send the public key to the other endpoint safely (client)
- encrypt some data with a random 256 bit password using AES256 CBC (server)
- encrypt the password using the public key (server)
- send the encrypted data to the client (server)
- decrypt the encrypted 256 bit password using the private key (client)
- decrypt the file
Basically I know how to handle AES encryption/decryption and the communication protocol, they're already implemented anyway, my issue is RSA, and I need hex format not base64 or pem as I'm working with sockets and sending and storing as binary data.
As discussed on IRC, the answer to the create_rsa_key question is addressed by https://stackoverflow.com/a/29589818/37923 - it's not meant to work that way.
pcszPassphrase is meant to be the password/phrase to protect the private key. It would be provided by the user, but isn't a requirement.