How do I pass runtime parameters to dotnet test on the command line?

Question

I have a set of unit tests in a .NET Core project and using a runsettings file. I am trying to setup Azure DevOps to do automated testing on my deployments. As part of this process, I need to override parameters from the run settings on the command line.

I have a runsettings file with the following section:

  <TestRunParameters>
    <Parameter name="ApiUrl" value="https://myurl..." />
  </TestRunParameters>

I have a static constructor that saves the TestContext like this:

        [ClassInitialize]
        public static void TestClassInitialize(TestContext context)
        {
            TestContext = context;
        }

I am retrieving settings from the TestContext with the following method:

        protected string GetStringSetting(string settingName)
        {
            return TestContext.Properties[settingName] as string;
        }

When I run the test with the runsettings file selected, it gets the TestContext and I see the ApiUrl entry is retrieved successfully.

Now, I want to set this parameter from the command line. I am using a command like:

    dotnet test <myproject.csproj> --configuration Release -- ApiUrl=https://newurl

I get an error that says the dictionary does not contain the key "ApiUrl" which indicates that the setting was not processed. From reading documentation, I thought that maybe I need to fully specify the name of the setting with TestRunParameters.ApiUrl. This gives me an XML error.

From everything I have read, I think I am doing this right and can't figure out what is wrong. I am using version 2.1.503 of the tools.

Can someone please give me guidance on how to make this work?

Answer 1

You could inject environment variables from your pipeline to overcome the limitations of the dotnet test command. By doing so, you will not need to deal with a temporary *.runsettings file to get your test parameters. You can set environment variables from your CI pipeline and then retrieve them in your tests at runtime.

Your YAML file could set environment variables:

run_tests:
  # Set environment variables as you need
  variables:
    FOO: "bar"

Your test can retrieve environment variables:

[Test]
public void Test()
{
    var foo = Environment.GetEnvironmentVariable("FOO"); //set to "bar"
}

Answer 2

This is currently impossible to achieve. Check those closed issues on github:

• Runsettings from command line (--) doesnt have capability to pass TestRunParameters
• dotnet test with TestRunParameters defined should pass parameters to the test runner

Your only option in here is to create another runsettings file with new values and pass it to dotnet test with --settings flag.

Answer 3

You can use the following argument syntax to do that:

dotnet test <myproject.csproj> --configuration Release -- TestRunParameters.Parameter(name=\"ApiUrl\", value=\"https://newurl\")

Source: https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md

Answer 4

You can't substitute the TestRunParameters from the command-line when using the DotNetCoreCLI@2 task or the dotnet test command, but you can create a temporary .runsettings file that contains the valid parameter value as part of your build pipeline in Azure Pipelines:

- powershell: |
   [xml]$doc = Get-Content Tests/Settings.runsettings
   $doc.RunSettings.TestRunParameters.ChildNodes.Item(0).value = 'https://newurl'
   $doc.Save("$(Build.SourcesDirectory)/Tests/UpdatedSettings.runsettings")
  displayName: 'Override TestRunParameters'

- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
   command: test
   projects: 'Tests/Tests.csproj'
   arguments: '-s $(Build.SourcesDirectory)/Tests/UpdatedSettings.runsettings'

- powershell: |
   Remove-Item $(Build.SourcesDirectory)/Tests/UpdatedSettings.runsettings
  displayName: Remove temporary .runsettings file

You'll find a complete YAML file and a sample test project on GitHub.