We have an automated testing framework which using CodedUI, links Ranorex (5.4.2), VSO 2013 (TFS) and VS 2013 on a dedicated automation server. It runs the tests in a sequence determined by a query in MTM which orders them in ascending test case ID order and has been running fine until recently:
An MTM Test Run Log shows them being scheduled in the correct ascending test case ID order. In this image you can see the test case ID sequence going from test case ID 607 on line 9 to test case ID 624 on line 10:
In the last week we've noticed that although the order of the tests in the query hasn't changed, the tests are being reordered when the test run is being prepared in MTM:
In this second image, you can see the sequence change to test case ID 607 on line 9 to test case ID 25285 on line 10. The sequence continues to jump back and forth in a very disorganised way. This is causing us issues because some of the later tests rely on test data generated in earlier tests and therefore fail when this data isn't available because the test is running earlier than it should do.
Nothing has changed in the sequence of the tests in the query and we haven't added any new tests, but our devs have recently installed Visual Studio 2013 Update 5 around the time this started happening.
Has anyone come across this issue before or know whether it could be due to this recent VS update? I have hunted and hunted but have not been able to find anything similar on any other site.
UPDATE I have investigated this further by doing a run of a smaller number of tests:
Again they go out of sequence during the test run at line 10 of the log. Line 13 contains the test that should have come at line 10:
I have run out of ideas on how to take this issue forwards, so any help would be hugely appreciated.