Unit Testing AppContext Switches

Question

Change Log

• Renamed question: AppContext Switch Failing In Unit Test (CryptographicException: Invalid Algorithm Specified)
• Added section at bottom explaining the issue in further detail after doing more investigating
• Added section at the bottom explaining a workaround I have implemented

I have a piece of code that I am writing unit tests for that signs some xml.

I noticed when developing the code that .NET 4.7.1+ will throw CryptographicException: Invalid algorithm specified. To get around this issue I used the following AppContext switches (sources: wiktor zychla, why am i getting invalid algorithm specified...).

I've put the following code in any app that calls my xml signing code:

AppContext.SetSwitch("Switch.System.Security.Cryptography.Xml.UseInsecureHashAlgorithms", true);
AppContext.SetSwitch("Switch.System.Security.Cryptography.Pkcs.UseInsecureHashAlgorithms", true);

When running in the wild, this code is functioning correctly, and my xml is being signed as expected.

I've written some unit tests for this code, it's my first real attempt at writing unit tests and I'm experiencing some weird edge cases.

try
{
    // Compute the signature.
    signedXml.ComputeSignature();
}
catch (CryptographicException cex)
{
    //Contains instead of == since this message contains newline characters...
    if (cex.Message.Contains("Invalid algorithm specified"))
    {
        string code = Environment.NewLine + "AppContext.SetSwitch(\"Switch.System.Security.Cryptography.Xml.UseInsecureHashAlgorithms\", true);" +
                        Environment.NewLine + "AppContext.SetSwitch(\"Switch.System.Security.Cryptography.Pkcs.UseInsecureHashAlgorithms\", true); ";

        throw new Exception($"Cryptographic exception was thrown, ensure that the following switches are set in the executing assembly: {code}", cex);
    }
    else
    {
        throw cex;
    }
}

So if the exception is thrown, I display a nice message saying please add the following code before calling this method.

My test class looks something like this:

[TestFixture]    
public class XMLSigningTests
{
    protected IXMLSigner _xmlSigner;
    protected byte[] _publicKey;
    protected string _password;

    [SetUp]
    public void Init()
    {
        _xmlSigner = new XMLSigner();
        _password = "test";
        _publicKey = GenerateTestCert(_password);
    }

    [TearDown]
    public void CleanUp()
    {
        _xmlSigner = null;
        _publicKey = null;
        _password = null;
    }

    [Test]
    public void SignXML_AppContextSwitchNotSet_Fail()
    {
        XMLDocument xmlDoc = GenerateXML();

        Assert.Throws<Exception>(() => { _xmlSigner.SignXML(xmlDoc, _publicKey, _password); });
    }

    [Test]
    public void SignXML_AppContextSwitchSet_Success()
    {
        AppContext.SetSwitch("Switch.System.Security.Cryptography.Xml.UseInsecureHashAlgorithms", true);
        AppContext.SetSwitch("Switch.System.Security.Cryptography.Pkcs.UseInsecureHashAlgorithms", true);

        XMLDocument xmlDoc = GenerateXML();

        var signedXml = _xmlSigner.SignXML(xmlDoc, _publicKey, _password); 

        Assert.IsFalse(signedXML == null);
        //More validation here
    }

    private XMLDocument GenerateXML()
    {
        return new XMLDocument("Some fancy xml here...");
    }

    private byte[] GenerateTestCert(string password)
    {
        //Use Pluralsight.Crypto to generate a self signed cert & export as .pfx
        return cert.Export(X509ContentType.Pfx, password);
    }
}

When I run SignXML_AppContextSwitchSet_Success which I'd expect to pass on its own, the test passes everytime (just me repeatedly running it). If I clean my solution and run all tests, the test always fails.

My xml generation class has no shared objects, it could infact be static.

The test fails because the cryptographic exception gets thrown (triggering my code and throwing the exception telling the dev (me) to add the app context switch).

Is there a way to verify that the config switches are being respected?

When running the code from my console tester app and in the wpf app I've built it into, the exception is never thrown. It is only thrown when I run all of the unit tests back to back. I would like these tests to run on an azure pipeline so it is important that it can run simultaneously as well as individually.

My thoughts on this exception are that maybe when I'm running all the tests together, the AppContext switches are not being honoured. I've tried to add checks before signing the xml:

AppContext.TryGetSwitch("Switch.System.Security.Cryptography.Xml.UseInsecureHashAlgorithms", out bool xmlSwitch);
AppContext.TryGetSwitch("Switch.System.Security.Cryptography.Pkcs.UseInsecureHashAlgorithms", out bool pkcsSwitch);

if (!xmlSwitch || !pkcsSwitch)
{
    throw new NotImplementedException("THESE VALUES SHOULD BE SET WTF");
}

Now you'd expect this code to fail my fail test (where the switches aren't set) and it does, you can see my crude NotImplementedException message in the test explorer. However the test i'm expecting to succeed throws a CryptographicException again...

System.Exception : Cryptographic exception was thrown, ensure that the following switches are set in the executing assembly: 
    AppContext.SetSwitch("Switch.System.Security.Cryptography.Xml.UseInsecureHashAlgorithms", true);
    AppContext.SetSwitch("Switch.System.Security.Cryptography.Pkcs.UseInsecureHashAlgorithms", true); 
      ----> System.Security.Cryptography.CryptographicException : Invalid algorithm specified.

As I said earlier, I'm very new to testing and I'm not sure if my approach to test design is wrong, if I'm just testing something that's spaghetti code under the hood so maybe I'm just doomed trying to test it or theres something I'm missing to allow correct testing of this case.

I'm currently using NUnit 3.12.0 + NUnit3TestAdapter 3.15.1 to write tests, although I was intially using MSTest 1.3.2 and experienced the same behaviour using both frameworks.

---

Update

I've discovered that the execution order of the following tests determines the success:

• SignXML_AppContextSwitchNotSet_Fail
• SignXML_AppContextSwitchSet_Success

I discovered this by reading posts such as (Unit test fails when running all, but passes when running individually) on tests failing when not running individually.

It got me thinking about the values being set, I want the following cases:

//Test I expect to fail
AppContext.SetSwitch("the switch..", false);

//Test I expect to pass
AppContext.SetSwitch("the switch..", true);

Eventhough I have already tested for these values being set correctly, are the underlying switches being updated? How can I cleanup these underlying objects?

---

Workaround / 'Nasty' Fix

I found the following issue on NUnit's GitHub (appdomain per test (isolation per test), for dealing with static classes). In this thread they discuss the option of a new AppDomain per test, this is exactly what I need in my case.

A ways down the thread jnm2 posts a util class to run code in a seperate AppDomain (code) (I won't post it here since this post's already massive). I took this util class and adapted my tests to work with it (you can't use any class variables or methods, meaning I had to copy my helper methods code).

Once I had implemented the util and adapted my tests to fit its criteria, the tests started passing both when I ran them individually & all at once.

Mission accomplished!

Answer 1

I had exactly the same problem! I came across this post and it motivated my solution!

Instead of running the tests in the separate AppDomain, I've ended up modifying the production code where the calls to the cryptographic library are made. I run the problematic calls in a separate AppDomain where I set the switches before proceeding.

This fixes all the tests as well as it enables the clients of my library not to mess with the switches.