I am trying to run a subset of our specrun tests as part of our VSTS release process. It works fine with the "Visual Studio Test" step with TestCategory=SmokeTest:
It finds 47 tests and executes them. When I use the same configuration using the "Run Functional Tests" like so:
I get the following output:
2017-05-22T13:12:01.3937396Z DistributedTests: Creating run for selected test assemblies with following parameters
2017-05-22T13:12:01.3937396Z DistributedTests: SourceFilter: **\*AcceptanceTests*.dll TestCaseFilter: TestCategory=SmokeTest
2017-05-22T13:12:01.3937396Z DistributedTests: Run title: Specflow Test Run
2017-05-22T13:12:01.3937396Z DistributedTests: is automated: True
2017-05-22T13:12:01.3937396Z ##[debug]DistributedTests: Test Machines Environment Url: dta://env/_Project/_apis/32858/201705221310348402
2017-05-22T13:12:01.3937396Z DistributedTests: test settings id : 8585
2017-05-22T13:12:01.3937396Z DistributedTests: build location: C:\TestDrop\AcceptanceTests-SpecFlow
2017-05-22T13:12:01.3937396Z DistributedTests: build id: 32858
2017-05-22T13:12:01.3937396Z DistributedTests: test configuration mapping:
2017-05-22T13:12:01.7387150Z DistributedTests: Test Run with Id 23835 Queued
2017-05-22T13:12:01.9137365Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:12:12.0518938Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:12:22.1867837Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:12:32.3244144Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:12:42.4917165Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:12:52.6228617Z DistributedTests: Test run '23835' is in 'InProgress' state.
2017-05-22T13:13:02.7562848Z DistributedTests: Test run '23835' is in 'Aborted' state.
2017-05-22T13:13:12.7604691Z ##[warning]DistributedTests: Test run is aborted. Logging details of the run logs.
2017-05-22T13:13:12.7604691Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test run is aborted. Logging details of the run logs.
2017-05-22T13:13:12.8654238Z ##[warning]DistributedTests: New test run created.
2017-05-22T13:13:12.8654238Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: New test run created.
2017-05-22T13:13:12.8654238Z Test Run queued for Project Collection Build Service (vsts).
2017-05-22T13:13:12.8654238Z
2017-05-22T13:13:12.8654238Z ##[warning]DistributedTests: Test discovery started.
2017-05-22T13:13:12.8654238Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test discovery started.
2017-05-22T13:13:12.8654238Z ##[warning]DistributedTests: Test Run Discovery Aborted . Test run id : 23835
2017-05-22T13:13:12.8654238Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Test Run Discovery Aborted . Test run id : 23835
2017-05-22T13:13:12.8654238Z ##[warning]DistributedTests: UnExpected error occured during test execution. Try again.
2017-05-22T13:13:12.8654238Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: UnExpected error occured during test execution. Try again.
2017-05-22T13:13:12.8654238Z ##[warning]DistributedTests: Error : SpecRun: discovered 101 from C:\TestDrop\AcceptanceTests-SpecFlow\.AcceptanceTests.dll
2017-05-22T13:13:12.8654238Z ##[debug]Processed: ##vso[task.logissue type=warning]DistributedTests: Error : SpecRun: discovered 101 from C:\TestDrop\AcceptanceTests-SpecFlow\.AcceptanceTests.dll
It is discovering 101 tests, the total rather than just the ones marked as SmokeTest and it's not even running them. It just bails out. I know both VSTS steps are using vstest.console.exe. Is there some other configuration I am missing?