Decrypting the openssl library generated cipher on the terminal and vice versa

Question

I have made an implementation using the OpenSSL library to encrypt a password. I am able to successfully encrypt and decrypt the password using the library in the code.

However, if I try to decrypt the cipher generated by the Library on a linux system on the terminal the decryption fails.

Encryption:

openssl enc -aes-256-cbc -base64 -salt -k <passphrase> -in plain.txt -out 
encrypt.txt

Decryption:

openssl enc -aes-256-cbc -base64 -salt -d -k <passphrase> -in encrypt.txt -out plain.txt

Please help.

I have removed salts so as to make the process simpler. I have generated sample base64 ciphers on the terminal and tried decrypting with Library, this fails. I have tried generating sample base64 ciphers with Library and tried decrypting it in the terminal, this fails, too!

size_t init_key_iv(const std::string& pass, const unsigned char* salt, unsigned char* key, unsigned char* iv ) {
size_t derived_key_size = 0;
  const unsigned char * pass_key = reinterpret_cast<const unsigned char*>( pass.c_str() );
  const size_t pass_key_len = pass.size();
  if(salt && key && iv && pass_key && pass_key_len > 0) {
    memset(key, 0, sizeof(key));
    memset( iv, 0, sizeof(iv));
    derived_key_size = EVP_BytesToKey(cipher_type, msg_digest_type, salt, pass_key, pass_key_len, 5, key, iv);
  }
 return derived_key_size;
}

void encrypt(const unsigned char* msg, unsigned char** encrypted_message, const size_t msg_len, const unsigned char *key, unsigned char *iv) {
    AES_KEY enc_key;
    AES_set_encrypt_key(key, 256, &enc_key);
    AES_cbc_encrypt(msg, *encrypted_message, msg_len, &enc_key, iv, AES_ENCRYPT);
}

void decrypt(const unsigned char* cipher, unsigned char** decrypted_msg, const size_t cipher_len, const unsigned char *key, unsigned char *iv ) {
  AES_KEY enc_key;
  AES_set_decrypt_key(key, 256, &enc_key);
  AES_cbc_encrypt(cipher, *decrypted_msg, cipher_len, &enc_key, iv, AES_DECRYPT);
}

int decode(const char* b64_msg, unsigned char** decode_msg, const size_t  decode_msg_len) {
  size_t bytes_decoded = 0;
  bytes_decoded = EVP_DecodeBlock(*decode_msg, (unsigned char *)b64_msg, strlen(b64_msg));
  return bytes_decoded;
}

int encode(const unsigned char* msg, const size_t msg_len, char** b64_msg) {
  size_t bytes_encoded = 0;
  if(msg && msg_len > 0 && b64_msg)  {
    bytes_encoded = EVP_EncodeBlock((unsigned char *) *b64_msg, msg, msg_len);
  }
  return bytes_encoded;
}

const int derived_key_size = init_key_iv(password, salt, key, iv_enc);
encrypt((unsigned char *)msg, &encrypted_message, strlen(msg), key, iv_enc);
const size_t bytes_encoded( encode((const unsigned char*)encrypted_message, strlen(reinterpret_cast<char*>(encrypted_message)), &base64_enc_str) );
const size_t bytes_decoded( CBase64::decode(cipher_base64, &cipher, cipher_len) );
decrypt(cipher, &decrypted_message, cipher_len, key, iv_dec);

Expectation is; the base64 ciphers generated by the Library should be decrypted in the openssl terminal and vice versa.

Answer 1

Your code is not complete, and there are several things not shown which could be wrong, but the things shown that are definitely or probably wrong are:

• you call EVP_BytesToKey with count=5, when commandline enc uses count=1. In addition you don't show what cipher_type and msg_digest_type are and they could be wrong; in particular, the default digest used for BytesToKey in commandline enc varies depending on the version of OpenSSL, and you didn't say what version(s) you are or will or might be using. Although specifying -md $hash overrides that default, which is a more robust and clearer solution.

• you don't show where the plaintext comes from, and in particular whether and how you've padded it. Commandline enc by default uses PKCS5/7 padding, and has an option to use no padding, but in that case the plaintext length must always be an exact multiple of 16 -- have you guaranteed that?

• you use strlen(ciphertext) as the length of the (raw = before base64) ciphertext; this is usually wrong. The ciphertext is effectively random bits, and can easily contain a byte with value 0, giving a strlen() that is too small, but if that doesn't happen it will not necessarily be followed or terminated by a 0 byte, giving a strlen() that is too large.

• you do not include in the base64 encoding the file header (aka 'magic') required by commandline enc when using salt. The code shown does not add the line breaks required by commandline enc file format, but that could be done elsewhere, and only matters if the values you encrypt (and want to decrypt) are or (ever) can be more than 31 bytes.

• Also, you call memset (key, 0, sizeof(key)) and the same for iv when they are pointers; this clears only the size of the pointer, 4 or 8 bytes on modern systems, not the object pointed to. But since these objects are promptly overwritten by BytesToKey, this mistake doesn't matter.

Anyway, here is a minimal-ish code that is complete and produces an output that is decryptable by commandline enc -aes-256-cbc -d -a -k $password with -md sha256 for versions below 1.1.0 where that is not the default. For convenience I have limited to 80 bytes input, but it should be obvious how to increase that if desired.

/* SO56447374 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/aes.h>
#include <openssl/bio.h>
#include <openssl/err.h>
#include <openssl/evp.h>
#include <openssl/rand.h>

void err (const char *label){
  fprintf (stderr, "%s:\n", label); ERR_print_errors_fp (stderr); exit (1);
}

int main (int argc, char**argv)
{
  ERR_load_crypto_strings(); 
  //OPENSSL_add_all_algorithms_noconf(); 

  const char * pass = argv[1];
  unsigned char salt [8], key [32], iv [16],
        plain [80], buffer [16+96], *cipher = buffer+16;
  int inlen = fread (plain, 1, 80-1, stdin), pad = 16-inlen%16U;
  AES_KEY aeskey;

  RAND_bytes (salt, 8);
  if( !EVP_BytesToKey (EVP_aes_256_cbc(), EVP_sha256(), salt, 
        (unsigned char*)pass, strlen(pass), 1, key, iv) ) err("BytesToKey");
  AES_set_encrypt_key (key, 256, &aeskey);
  memset (plain+inlen, pad, pad); // PKCS5/7 
  AES_cbc_encrypt (plain, cipher, inlen+pad, &aeskey, iv, AES_ENCRYPT);
  memcpy (buffer+0, "Salted__", 8); memcpy (buffer+8, salt, 8);
  BIO *bio1 = BIO_new (BIO_f_base64()); // does b64 with linebreaks (by default)
  BIO_push (bio1, BIO_new_fp (stdout, BIO_NOCLOSE));
  BIO_write (bio1, buffer, 16+inlen+pad);
  BIO_flush (bio1);
  BIO_free_all (bio1);
  return 0;
}