multiple asynchronous tests and expectation

Question

I have multiple tests and each test is testing the same asynchronous method for different results with given parameters.

I found out for asynchronous tests we have to declare an expectation, wait for expectation, and fulfil the expectation. This is fine. Each test works out correctly when done separately, but when I try to run the whole test class some tests pass and others crash or fail when they run and pass normally.

I've looked all over online for "swift 3 multiple tests with expectation" and everyone who explains expectation only ever has an example in one test method. Is it not possible to have expectations in multiple methods in the same class?

An example of a test is as follows:

func testLoginWrongUsernameOrPasswordFailure() {
  let viewModel = LoginViewModel()
  let loginAPI = APIManager()
  let expect = expectation(description: "testing for incorrect credentials")
        
  viewModel.loginWith(username: "qwerty", password: "qwerty", completion: { loginCompletion in
            
      do {
        try loginCompletion()
          XCTFail("Wrong Login didn't error")
          expect.fulfill()
        } catch let error {
          XCTAssertEqual(error as? LoginError, LoginError.wrongCredentials)
          expect.fulfill()
        }
      })
        
      waitForExpectations(timeout: 10) { error in
        XCTAssertNil(error)
      }
}

As far as I'm aware, this is the correct use of expectation and each test follows the same pattern

As requested by Rob I will provide an MCVE here https://bitbucket.org/chirone/mcve_test The test classes use a mock API Manager but when I was testing with the real one the errors still occurred.

As an explanation for the code, the view-model communicates with a given API manager who invokes a server and gives back the response to the view-model for him to interpret the errors or success.

The first test tests for empty fields, something that the view-model validates rather than the APIManager. The second test tests for incorrect username and password The third test tests for valid username and password

The three tests run separately will run fine, however when the whole file is run I will get a SIGABRT error with the following reasons:

XCTAssertEqual failed: ("Optional(MCVE.LoginError.wrongCredentials)") is not equal to ("Optional(MCVE.LoginError.emptyFields)") -

*** Assertion failure in -[XCTestExpectation fulfill], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-12124/Sources/XCTestFramework/Async/XCTestExpectation.m:101

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for testing for empty fields.'

The SIGABRT happens usually on the second test method and if you hit play then it fails on one of the XCTest methods claiming the error it got is not the error it was expecting.

I hope the MCVE helps explain my problem.

Answer 1

Is it possible to wait for multiple expectations; yes. Here is a signature for an XCTestCase method that shows this.

func wait(for: [XCTestExpectation], timeout: TimeInterval)

There is a version that also makes sure that the expectations are fulfilled in the same order as they appear in the for: array.

See the documentation provided by Apple in XCode->Window->Documentation and API Reference, then search for XCTestCase.

Answer 2

Refactored the code as given below.

func testLoginWrongUsernameOrPasswordFailure() {
  let viewModel = LoginViewModel()
  let loginAPI = APIManager()
  let expect = expectation(description: "testing for incorrect credentials")

  viewModel.loginWith(username: "qwerty", password: "qwerty", completion: { loginCompletion in

      do {
        try loginCompletion()
        XCTFail("Wrong Login didn't error")

      } catch let error {
        XCTAssertEqual(error as? LoginError, LoginError.wrongCredentials)
      }
      expect.fulfill()
   })

  waitForExpectations(timeout: 10) { error in
    XCTAssertNil(error)
  }
}

If you are still getting the following crash, that means your completion handler for the asynchronous code is calling multiple time. And there by invoking expect.fulfill() multiple times. Which is not allowed.

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for testing for empty fields.'

For an expectation fulfill() should call only once. If there are some rare scenarios and you need to invoke expect.fulfill() more than once then set the following property.

expectedFulfillmentCount

Please refer the following link https://developer.apple.com/documentation/xctest/xctestexpectation/2806572-expectedfulfillmentcount?language=objc

Answer 3

If you have multiple tests (methods) in one XCTestCase don't use

let expectation = expectation(description: "")

Instead, use

let expectation = XCTestExpectation(description: "")

The self.expectaion() is shared between XCTestCase tests. In some cases, it brings to weird behavior. For example, you can get an API violation error even if you fulfill the expectation zero times.