Cryptoapi sign/verify not working on Windows 8.1

Question

We have an application that signs and verifies messages with SHA1 hashes using CryptoApi. It has worked perfectly for many years under WindowsXP up to Windows 8. It no longer works on Windows 8.1. CryptSignHash fails with error code 87 (Invalid Parameter). CryptVerifySignature does not fail but returns NTE_BAD_SIGNATURE (for valid signatures created on Windows 8). We have tested everything we can .. it works on Windows 8 and below, fails on Windows 8.1.

Do you have any ideas on how to debug this further? We exported the imported public and private keys again and verified that they are correct. We skipped using 'our' keys and generated new keys -> signing fails also with error 87 We generated new keys and encrypted/decrypted a message with them using RSA_FULL and DES -> no problem, works as expected. We checked the version of the RSA_FULL provider. It is 2.0 on both Windows 8 and Windows 8.1. We tried explicitly specifying the provider name: Microsoft Base Cryptographic Provider v1.0

Does signing work for any of you on Windows 8.1.?

Is there anything else that is new on Windows 8.1. that could prevent signing from working? Has something changed with respect to the providers or algorithms that we should know?

The application is written in Delphi, uses approximately the following flow:

//Setup crypto provider 
CryptAcquireContext(@fhCryptProv, nil, nil,  cptRSAFull, [ccVerify, ccMachineKeySet]); 

//Create a hash structure
CryptCreateHash( fProvider.GetProviderHandle, chtSHA1, 0, 0, @fhHash);

//Import the private key for signing
CryptImportKey( fProvider.GetProviderHandle, @buff[0], len, 0, CRYPT_EXPORTABLE, @fKey);

//Hash the message
CryptHashData(fhHash, @aPlainText[1], length(aPlainText) * 2, 0);

//Sign the message
CryptSignHash(fhHash, AT_SIGNATURE, nil, CRYPT_NOHASHOID OR CRYPT_X931_FORMAT, @buff, @len);

Answer 1

CryptSignHash fails on Windows 8.1 when CRYPT_X931_FORMAT flag is set. This looks like a Windows 8.1. bug.

This bug can be reproduced by running the MSDN Sample Code from the url below, using CALG_SHA1 instead of CALG_MD5 for hashing and setting the flags in CryptSignHash to CRYPT_NOHASHOID | CRYPT_X931_FORMAT; leaving the flags at 0 or just at CRYPT_NOHASHOID works on the systems I tested with.

I never had any problems with these flags on WindowsXP to Windows 8.0 systems.

MSDN Sample code URL: http://msdn.microsoft.com/en-us/library/windows/desktop/aa382371%28v=vs.85%29.aspx