Running Xcodes UITesting on Jenkins never fails a Test

Question

I have built a simple UITesting framework for one of my apps built in Xcode. 2 of the tests pass and one purposefully fails:

XCTAssertTrue(false)

I am using the Jenkins Xcode plugin and am using the post-built action: Publish JUnit test result report.

Jenkins successfully launches the simulator and runs all 3 tests. It also successfully picks up the failure in the logs:

Failing tests:
-[LightAlarmUITests testFailingTest()]
** TEST FAILED **

However, the Test Results Analyzer (plugin installed) shows all 3 tests are passing. When I inspect the test-results/*.xml file I see the following:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuite failures="1" errors="0" hostname="Charlies-MacBook-Pro.local" name="LightAlarmUITests" tests="3" time="21.0" timestamp="2016-12-20T16:24:33.125Z">
<testcase classname="LightAlarmUITests" name="testFailingTest" time="6.372"/>
<testcase classname="LightAlarmUITests" name="testShowSettingsPage" time="7.167"/>
<testcase classname="LightAlarmUITests" name="testShowSoundAlarmsPage" time="7.594"/>
</testsuite>

As you can see in the xml, it is not marking individual test cases as pass or fail, but instead marking a failure against the entire test suite.

Does anyone know how to mark pass/fails against individual test cases?

Answer 1

There is a pull request open to fix this issue.

https://github.com/jenkinsci/xcode-plugin/pull/75

Answer 2

The reason behind this is Xcode 8 changed the way of XML of UI test report. JUnit plugin is not able to parse this XML to generate a test report. I have used XCPretty mediator that converts Xcode 8 XML report into JUnit supported XML. You can find more details at https://blog.talentica.com/2017/04/04/use-xcode-8-with-jenkins/