RC2 decryption from wincrypt API to Go

Question

I want to rewrite old C code from 1990 to Go. But the pain point here is migration of cryptographic algorithms. I have the following Cpp code that successfully decrypts the cipher into plain text.

#include <windows.h>
#include <wincrypt.h>
#include <string>
#include <iostream>

using namespace std;

int main() {
    auto name = L"aaaa";

    HCRYPTPROV hProv;

    if (!CryptAcquireContext(&hProv, name, MS_DEF_PROV, PROV_RSA_FULL, 0) &&
        !CryptAcquireContextW(&hProv, name, MS_DEF_PROV, PROV_RSA_FULL, CRYPT_NEWKEYSET) &&
        !CryptAcquireContextW(&hProv, name, MS_DEF_PROV, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET) &&
        !CryptAcquireContextW(&hProv, name, MS_DEF_PROV, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET | CRYPT_NEWKEYSET)) {
        cout << "fail" << endl;
        exit(1);
    }

    HCRYPTHASH hHash;

    if (!CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
        cout << "fail" << endl;
        exit(1);
    }

    const BYTE* pwd = reinterpret_cast<const BYTE*>("-+ REDACTED +-");

    if (!CryptHashData(hHash, pwd, 14, 0)) {
        cout << "md5 failure" << endl;
        exit(1);
    }

    HCRYPTKEY hKey;

    if (!CryptDeriveKey(hProv, CALG_RC2, hHash, 0, &hKey)) {
        cout << "failure" << endl;
        exit(1);
    }

    unsigned char cipher[] = {52, 54, 253, 199, 131, 110, 202, 15, 185, 107, 71, 244, 150, 171, 220, 6, 183, 86, 234, 252, 242, 84, 156, 200};

    DWORD len = 24;

    if (!CryptDecrypt(hKey, 0, TRUE, 0, cipher, &len)) {
        printf("%x\n", GetLastError());
        exit(1);
    }

    for (int i = 0; i < len; i++) {
        printf("%d, ", cipher[i]);
    }
}

and I have the following Go code:

package main

import (
    "crypto/md5"
    "fmt"

    "github.com/dgryski/go-rc2"
)

func main() {
    pwd := "-+ REDACTED +-"
    hash := md5.Sum([]byte(pwd))

    alg, err := rc2.New(hash[:], 128)
    if err != nil {
        panic(err)
    }

    cipher := []byte{52, 54, 253, 199, 131, 110, 202, 15, 185, 107, 71, 244, 150, 171, 220, 6, 183, 86, 234, 252, 242, 84, 156, 200}

    result := []byte{}
    dst := make([]byte, 8)

    for i := 8; i <= len(cipher); i += 8 {
        alg.Decrypt(dst, cipher[i-8:i])
        result = append(result, dst...)
    }

    fmt.Println(result)
}

run online: https://go.dev/play/p/veRMRShmtnw

The output of them are different. While the CPP version able to decrypt correctly.

My assumption is, that `CryptDeriveKey` harden the initial hash password with additional values and as a result encryption key gets altered.

Answer 1

To understand the encryption/decryption in the C code, it is helpful to know the parameters used. With CryptGetKeyParam() the parameters can be output. This gives:

• KP_KEYLEN, KP_EFFECTIVE_KEYLEN: Key length and effective key length are each 40 bits long.
• KP_SALT: As salt a zero salt is used (11 times 0x00).
• KP_MODE: As mode CBC is applied (value CRYPT_MODE_CBC).
• KP_BLOCKLEN: The block length is 64 bits long.
• KP_IV: As IV a zero IV is used (8 times 0x00, corresponding to the block length).
• KP_PADDING: As padding PKCS#7 is applied (value: PKCS5_PADDING).

These settings must be applied when implementing the Go code.

---

The crucial point about the derived RC2 key is that it is generally composed of two parts. The first has the length KP_KEYLEN, the second is the salt, see here:

• Regarding the first key part: It has already been mentioned in the comments that the MS Base Cryptographic Provider you are using (MS_DEF_PROV) supports a size of 40 bits for RC2 by default. This length refers to the first key part.

  To determine the algorithm that is applied to create the key, it is useful to determine the key first. The key can be exported with CryptExportKey() if the CRYPT_EXPORTABLE flag is set in CryptDeriveKey(). The format is BLOBHEADER (8 bytes)|DWORD key length (2 bytes)|key material (see here).
  The export proves that this key part (i.e. key material) corresponds to the first 40 bits of the MD5 hash of the input key (pwd), i.e. CryptDeriveKey() generates the hash of the specified digest from the input key and returns the preceding KP_KEYLEN bits of the hash as first key part.

• The second key part is the salt. This can be controlled by various CryptDeriveKey() flags. If CRYPT_CREATE_SALT is set, a salt is applied, if this flag is not set, a salt consisting of 0x00 values is used (this corresponds to your case). If no salt is to be applied, CRYPT_NO_SALT must be set.

Example (assuming that the flags are set as in the C code):
If the input key is some test key, the MD5 hash of it is 0x9bf00ebbb4522bbf2a6209f00372b0a7. From this the first 5 bytes (40 bits) are used and the 11 bytes zero salt is appended, i.e. the final key is 16 bytes (128 bits) long and is 0x9bf00ebbb40000000000000000000000.

---

In addition to the specifics about the key, the following must be implemented in the Go code (according to the above parameters):

• The effective key length is 40 bits (as opposed to the key length, which is 128 bits because of the salt).

• As mode CBC is used, as IV a zero IV.

• As padding PKCS#7 is applied.

With this information, a possible Go implementation is:

import (
    "crypto/cipher"
    "crypto/md5"
    "encoding/hex"
    "fmt"

    "github.com/dgryski/go-rc2"
    "github.com/zenazn/pkcs7pad"
)

func main() {
    ciphertext, _ := hex.DecodeString("596914c6020c9dbfc193e9e588a380730add8e1a69a4994ec57362dd1ad1d37a00932a64a385af6d8c5234b0c36d50c4")
    iv, _ := hex.DecodeString("0000000000000000")

    key := "some test key"
    hash := md5.Sum([]byte(key))

    finalKey := make([]byte, 16)
    copy(finalKey[:], hash[:5])
    fmt.Println(hex.EncodeToString(finalKey)) // 9bf00ebbb40000000000000000000000

    plaintextPadded := make([]byte, len(ciphertext))
    alg, _ := rc2.New(finalKey, 40)
    mode := cipher.NewCBCDecrypter(alg, iv)
    mode.CryptBlocks(plaintextPadded, ciphertext)
    plaintext, _ := pkcs7pad.Unpad(plaintextPadded)

    fmt.Println(hex.EncodeToString(plaintext)) // 54686520717569636b2062726f776e20666f78206a756d7073206f76657220746865206c617a7920646f67
    fmt.Println(string(plaintext))             // The quick brown fox jumps over the lazy dog
}

The test data can also be decrypted with the C code:

...
const char* pwd = "some test key";
CryptHashData(hHash, reinterpret_cast<const BYTE*>(pwd), strlen(pwd), 0);
...
unsigned char cipher[] = { 0x59, 0x69, 0x14, 0xc6, 0x02, 0x0c, 0x9d, 0xbf, 0xc1, 0x93, 0xe9, 0xe5, 0x88, 0xa3, 0x80, 0x73, 0x0a, 0xdd, 0x8e, 0x1a, 0x69, 0xa4, 0x99, 0x4e, 0xc5, 0x73, 0x62, 0xdd, 0x1a, 0xd1, 0xd3, 0x7a, 0x00, 0x93, 0x2a, 0x64, 0xa3, 0x85, 0xaf, 0x6d, 0x8c, 0x52, 0x34, 0xb0, 0xc3, 0x6d, 0x50, 0xc4 };
DWORD len = sizeof(cipher);
CryptDecrypt(hKey, 0, TRUE, 0, cipher, &len);
...

which shows that both codes are functionally identical.