Implementing challenge response authentication using RSA

Question

I'm trying to implement a challenge response protocol using RSA with one side being a WCF service and the other a Windows Phone 7.1 client. I'm using the native implementation in .NET

The protocol says that

1. the client sends a request to the server

2. the server sends a challenge to the client

3. the client encrypts the challenge using his private key and sends it to the server

4. the server decrypts the value using the client's public key and compares it to the challenge

but I can't get step 3 to work. when I try to decrypt I get a CryptographicException Unknown Error 80007001 on this line:

decryptedChallengeBytes = rsaProvider.Decrypt(challengeBytes, false);

I have tried both true and false. What should I do?

Answer 1

I used the VerifyHash method. It takes a hashed input and signs it using the private key which is exactly what I needed to do.