TFS vNext build Test Impact Analysis - always running all tests from some projects in the solution

Question

I have a C# Visual Studio 2017 solution with multiple projects and test projects, a vNext build pipeline with VS Test task. The "Run only impacted tests" option is enabled. Clean options are unchecked.

I start a build with no changesets (no code changes). In the test agent's workspace, I can verify that there is no rebuild occurring: the project references, the Copy Local's, pre/post build events, Copy if Newer's are all good. No assemblies are replaced after the build.

Tests from some projects are not run (which is correct), but tests from some projects are always run.

The project properties of these test projects seem identical.

Does anybody have a suggestion on where to look?

Answer 1

Please make sure no failed tests in previous build/test, and no new files generated during last build to effect current build/test.

Please refer to below links to know about what are the tests that get automatically selected and policies that can condition test selection for Test Impact Analysis .

• Accelerated Continuous Testing with Test Impact Analysis – Part 1
• Accelerated Continuous Testing with Test Impact Analysis – Part 2

Test Impact Analysis (TIA) will look at an incoming commit, and select the set of relevant tests – these will have 3 components

(1) The existing tests impacted by the incoming commit.

(2) Additionally, it will also select previously failing tests – if not, then over the course of several commits some earlier failing test case might just get lost … Therefore, TIA will keep track of tests that failed in the previous build and include them in the selection.

(3) It will also include newly added tests – what if your commit contains new tests? Such tests could uncover product bugs, right? So, TIA will select newly added tests as well.

TIA has:

(1) a robust test selection – this includes existing impacted tests, previously failing tests and newly added tests.

(2) safe fall back – for commits and scenarios that TIA cannot reason about, it will safely fall back to running all tests (you can see that mentioned in the logs). TIA is currently scoped to only managed code, and single box topology. So for e.g. if the code commit contains changes to HTML/CSS files, it cannot reason about them and will fall back to running all tests.

(3) configurable overrides – you can run “all” tests at a configured periodicity.

Answer 2

After days of inspection I've found the cause of this issue. I'm sharing this so maybe it helps others.

VS Test task (actually the vstest.console.exe in it) publishes results to a trx file and a number of text files that contains the dependencies of the tests.

1) If you have tests having the same (case insensitive) name, this publishing works unreliably. As it turned out, we had tests like this (i.e. TestThisWhenStringIsTrue and TestThisWhenStringIsTRUE).

2) Full names of the test methods (..) should not exceed 256 characters. When this is the case, the publishing works again unreliably.

After refactoring the tests the TIA started working. I hope this helps.

Note: Before coming to this step, you first need to have your incremental build running of course.