DevOps error: "Validating stage:The selected stage does not have the right version or settings of the Visual Studio Test task to run tests."

Question

I am getting the error "Validating stage:The selected stage does not have the right version or settings of the Visual Studio Test task to run tests. Learn more" when trying to run tests from a test plan on Azure DevOps.

Learn more links to this article:

• https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops

We performed quiet a bit of searching, but none of them worked for us. This could be a simple setting that we are overlooking. Your help is greatly appreciated.

Complete steps to reproduce the problem:

1. Login to your DevOps account
2. Create a test plan, call it 'TestPlan"
3. Add a single test to the plan, and note the test case ID (let's say 65)
4. On your local machine, open VS 2019 using Git as your source control
5. Create an MS Test project using C# and .net 4.6.2, call it "TestProject"
6. Add a single test to the project, name it TC65_UnitTest1, and execute the test

using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace TestProject
{
    [TestClass]
    public class TC65_UnitTest1
    {
        [TestMethod]
        public void TC65_TestMethod1()
        {
            // Test 1
            Assert.IsTrue(true);
        }
    }
}

6. From visual studio Test Explorer, associate the test case "TC65_TestMethod" with test case ID 65.
7. Push your repo to Azure
8. From Azure, create a build pipeline follwoing the steps in this article "https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops"
9. My build yml file is shown bellow
10. Run the build pipeline
11. Create a release pipeline with 2 tasks
12. Run the release pipeline
13. Open the test plan on Azure notice that test case ID 65 passed
14. Still on the test plan, click on "Execute"
15. Select "Run with options"
16. On the dialog pop up, select "
17. Click "Run"
18. Expected the test case to show pass. Instead, I get the following error

#AzureDevops.yml
trigger: none

pool:
  vmImage: 'windows-latest'
  demands: vstest

steps:
- task: DotNetCoreCLI@2
  displayName: '1. Build Solution'
  inputs:
    command: 'build'
    projects: '**/*.csproj'
- task: VSTest@3
  displayName: '2. Run Tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        !**\*
        **\bin\Debug\net462\TestProject.dll
      searchFolder: '$(System.DefaultWorkingDirectory)'
      codeCoverageEnabled: false  # Disable code coverage
- task: PublishBuildArtifacts@1
  displayName: '3. Publish Build Artifacts'
  inputs:
    PathtoPublish: 'TestProject'

I followed the article:

• https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops"

I was expecting to execute a test case from the test plan on Azure DevOps, but I encountered an error.

Answer 1

To Run Automated Test Plans using a Test Plan in Azure DevOps You will need a release pipeline that was created from the Run automated tests from Test Manager template to run tests from test plans in Azure Test Plans.

The Template defines a build stage with a Test Task. The Test Task should have setting as defined below, to allow for the stage to be invoked using pipeline parameters.

Edit:

Adding Additional Screenshot of How the release pipeline receives test.runid as pipeline variable when invoked using run with options.

Answer 2

I've done my investigations, and I've managed to run it on Azure DevOps (October 2023). In your release pipeline you must:

• Set up an agent job, a custom agent with vs2015 or vs2017
• Select tests using: test run

Otherwise, it won't work.