When I run android CTS full test using below command
run cts --plan CTS
Every time it shows different result for some of the packages, I mean some packages some tests passes/fails randomly every time I re-run full test. But when I run package individually (The package in which some tests failed), all the tests passes in it.
Why I am seeing this behavior?
Environment: OS: Android L CTS version: 5.1_r7
On
Some cts tests involve specific timeouts set for some event to occur. For example if you are running cts test related to data calls like turning mobile data off/on and timeout to get mobile data connected is set to 10 seconds, then some time this test will pass and sometime it will fail. In this case, increasing that timeout will resolve this issue.
Regarding issue of test case failing when running multiple packages, there could be possibility that test case before failed one has not set device in a neutral/original state for next test. It is a good practice to revert all changes made during a test while exiting a test case.
It happen some time some test failed randomly because some time that test condition is satisfied some time not and some time because of timeout test may fail.