I'm running KIF test suites on Amazon Device Farm. I selected XCTest when submit the job. I observed an unexpected run flow that is different from running the test locally from Xcode.
KIF version is 3.5.1, Xcode version is 8.2.
As we know, KIFTestCase subclasses XCTestCase to add setup and teardown steps that can be used to execute KIF test steps."beforeAll" and "afterAll" which run once before and after all tests and "beforeEach" and "afterEach" which run before and after every test. So it I have a test suite with 3 testcases, the flow will be:
beforeAll beforeEach test1 afterEach beforeEach test2 afterEach beforeEach test3 afterEach afterAll
When I run the tests locally with Xcode or on Jenkins, it DOES follow the above flow sequence. However when I submit the job to Amazon Device Farm, I can see it from the log as well as the video that it runs as the following:
beforeAll beforeEach test1 afterEach afterAll beforeAll beforeEach test2 afterEach afterAll beforeAll beforeEach test3 afterEach afterAll
i.e each testcase is run totally from SCRATCH: login, setup DB, recreated prep instances for testing; test; cleanup and logout.
Since the beforeAll usually takes about more than 1min and each test takes only a few sec, this flow sequence made the test run MUCH more time consuming and therefore expensive.
I'm wondering if anyone is seeing the same thing and whether this is the expected behave on Amazon Device Farm for XCTest ?
This is expected behavior in AWS Device Farm. The service runs each test in isolation from every other test.