Tests pass in Xcode but fail on Circle CI

Question

Question: For this pull request, the tests do not pass on CircleCI but the tests pass locally. Why?

---

The CircleCI test output shows failures for all FBSnapshotTestCase tests. For example:

✗ testAdjustsFontSizeToFitWidth, ((comparisonSuccess__) is true) failed - Snapshot comparison failed: Error Domain=FBSnapshotTestControllerErrorDomain Code=1 "Unable to load reference image." UserInfo=0x7f85f36b0a50 {NSLocalizedFailureReason=Reference image not found. You need to run the test in record mode, NSLocalizedDescription=Unable to load reference image., FBReferenceImageFilePathKey=/Users/distiller/TTTAttributedLabel/Example/TTTAttributedLabelTests/ReferenceImages_32/TTTAttributedLabelTests/[email protected]}

[…]

✗ testAttributedTruncationToken, ((comparisonSuccess__) is true) failed - Snapshot comparison failed: Error Domain=FBSnapshotTestControllerErrorDomain Code=1 "Unable to load reference image." UserInfo=0x7f85f35b06d0 {NSLocalizedFailureReason=Reference image not found. You need to run the test in record mode, NSLocalizedDescription=Unable to load reference image., FBReferenceImageFilePathKey=/Users/distiller/TTTAttributedLabel/Example/TTTAttributedLabelTests/ReferenceImages_32/TTTAttributedLabelTests/[email protected]}

However, the same tests pass locally:

On CircleCI the tests use the ReferenceImages_32 directory, which does not exist:

/Users/distiller/TTTAttributedLabel/Example/TTTAttributedLabelTests/ReferenceImages_32/TTTAttributedLabelTests/[email protected]

However, the images do exist in ReferenceImages_64. I expect the _64 directory to be used because the circle.yml file specifies to use the iPhone 6 simulator:

dependencies:
  pre:
    - xcrun instruments -w "iPhone 6 (8.3 Simulator)" || exit 0
  override:    
    - sudo gem install cocoapods xcpretty obcd -N
    - pod install --project-directory=Example
test:
  override:
    - set -o pipefail && xcodebuild -workspace 'Example/Espressos.xcworkspace' -scheme 'Espressos'
      -sdk iphonesimulator -destination "platform=iOS Simulator,name=iPhone 6"
      GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES GCC_GENERATE_TEST_COVERAGE_FILES=YES clean test | xcpretty -c
      --report junit --output ${CIRCLE_TEST_REPORTS}/junit.xml

• What causes this?
• How can I get the tests passing on CircleCI?
• Possibly related: this FBSnapshotTestCase issue

Answer 1

Have you tried running the build without the cache? This often fixes these kinds of issues for me. To build without cache, just click the button at the top right.