How to rerun failed tests right after they fail in Azure DevOps release pipeline

348 Views Asked by At

I have a Azure DevOps release pipeline, which has VSTest@2 task configured to execute my selenium tests. It reruns the failed tests but it waits till the end of the execution of all the tests and then re-executes the failed tests. I have written some sequential tests which needs to be run one after another as the test data created by one test will be used by subsequent tests. I want the failed tests to be rerun immediately after it fails instead of being re-executed at the end. For example, if I have 5 tests and test#2 fails, I don't want the rest of tests to be executed till the failed test has been rerun and passed or the max rerun attempt has been reached. Is there any way to acheive this. Tech used: Selenium with C#, nUnit

The task is already configured to rerun failed tests but it does so after the execution of all the tests is complete. Task yaml variables: Category: 'Regression'

steps:

  • task: VSTest@2 displayName: 'VsTest - testAssemblies' inputs: testAssemblyVer2: '**/Automation.dll' testFiltercriteria: 'Category=$(Category)' failOnMinTestsNotRun: false rerunFailedTests: true continueOnError: true
2

There are 2 best solutions below

0
On BEST ANSWER

Update on this: nunit has a "Retry" attribute, where you can give no. of retries you want. If there is an assertion failure in the test, the test execution will be retried at that instant and the execution won't wait for all the tests to get executed once.

0
On

I understand your requirement with VSTest@2 task. Unfortunately, there is no built-in option to do that in the current version of the task.

The rerunFailedTests feature only works at the end of the test run, not during the test run.

One possible workaround is that you can split your tests into different test assemblies or test categories, and run them in separate VSTest@2 tasks. You can rerun if the any test fails.

If you would like the feature to rerun the test if it fails, you can raise a user voice ticket in link here.