Running BenchmarkDotNet within XUnit

Question

I am using .NET Core 3.1 in my project (web api, VS2019) and XUnit 2.4.1. Recently I was thinking about adding some performance tests and I came accross this library - BenchmarkDotNet. Since I've already been using XUnit for other tests I wanted to run BenchmarkDotNet from within the XUnit [Fact]s.

I found this post where it is explained that shadow copies of assemblies must be turned off for xunit. So I tried following:

1. To keep it simple I have created fresh new .net core 3.1 console application project with sample method I wanted to benchmark:

[SimpleJob(RuntimeMoniker.NetCoreApp31)]
[MinColumn, MaxColumn, MedianColumn, KurtosisColumn]
[HtmlExporter]
public class TestScenarios
{
    [Params("test")]
    public string TextToHash { get; set; }

    [Benchmark]
    public string CalculateSha256()
    {
        var engine = SHA256.Create();
        var hash = engine.ComputeHash(Encoding.ASCII.GetBytes(TextToHash));
        return Encoding.ASCII.GetString(hash);
    }
}

Then in Program.cs I have:

class Program
{
    static void Main(string[] args)
    {
        BenchmarkRunner.Run<TestScenarios>();
    }
}

When compiling and runnning the application in Release I verified that BenchmarkDotNet.Artifacts folder with log and benchmark output files were correctly created.

2. I've added XUnit project to the solution and created simple [Fact] method:

public class DotNetBenchmarkTest
{
    [Fact]
    public void TestMethod()
    {
        BenchmarkRunner.Run<TestScenarios>();
    }
}

Also building in Release configuration and running this method from test explorer will create BenchmarkDotNet.Artifacts folder in bin/Release/dotnetcoreapp3.1/ but log files are empty.

3. I've also tried to add xunit.runner.json file with following contents to the root of my XUnit project:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "shadowCopy": false,
  "methodDisplay": "classAndMethod",
  "diagnosticMessages": true
}

but seems to have no effect on the behavior - I verified that file xunit.runner.json is copied to the output bin directory (Properties->Copy to Output Directory->Copy always).

4. I also installed xUnit.net Console Runner v2.4.1 and tried to run the tests from the console:

xunit .\path\assembly-name.dll -noshadow

Again no luck. The option -noshadow I found at this github link. Seems like the option was available in the older version but is not listed my help output for xunit console runner v2.4.1 but I gave it a try.

So my question is if it is possible to get DotNetBenchmarkTest working altogether with XUnit and if so, can you point me what am I doing wrong?

Answer 1

I am running BenchmarkDotNet benchmarks within XUnit tests.

Make sure that your project is referencing these NuGet packages

• xunit
• xunit.runner.visualstudio
• Microsoft.NET.Test.Sdk

Then you can run the xunit tests from Visual Studio or with dotnet test via command-line since this is .NET Core project.

xunit.console.runner is capable of running .NET Framework projects.

Check here and here for more explanation.

Answer 2

I have created a wrapper test fixture that runs the benchmarks, collects the output and prints the summary at the end:

public class Benchmarks
{
    private readonly ITestOutputHelper output;

    public Benchmarks(ITestOutputHelper output)
    {
        this.output = output;
    }

    [Fact]
    public void Run_Benchmarks()
    {
        var logger = new AccumulationLogger();

        var config = ManualConfig.Create(DefaultConfig.Instance)
            .AddLogger(logger)
            .WithOptions(ConfigOptions.DisableOptimizationsValidator);

        BenchmarkRunner.Run<FooBenchmarks>(config);

        // write benchmark summary
        output.WriteLine(logger.GetLog());
    }
}

[MemoryDiagnoser]
public class FooBenchmarks
{
    [GlobalSetup]
    public void Setup()
    {
        ModuleInitializer.Initialize();
        ...
    }

    [Benchmark]
    public async Task ProcessRequest_Benchmark()
    {
        ...
    }
}