ctest reports success for test with exit status 1

Question

I have a project with unit tests and when I run ctest (cmake version 3.18.2 on macos), success is reported for all tests. However, if I run one of the tests by itself, it has exit status 1. As far as I know, this shouldn't happen, so what is causing this and how can I fix it?

The issue was a small careless mistake, but for the benefit of others running into the issue, I provided an answer for which the rest of the question can be skipped.

Unfortunately, I cannot reproduce this behavior with a smaller minimum working example. I will try to provide as much relevant information as possible, please let me know if I am missing something. Here is the code of the unit test:

#include "alignment_reader.h"

#define CATCH_CONFIG_MAIN
#define CATCH_CONFIG_COLOUR_NONE
#include "catch.h"

#include "string_conversions.h"

#include "exceptions.h"

namespace paste_alignments {

namespace test {

namespace {

SCENARIO("Test correctness of AlignmentReader::FromFile.",
         "[AlignmentReader][FromFile][correctness]") {

  GIVEN("The name of a valid input file.") {
    std::string input_file{"test_data/valid_alignment_file.tsv"};

    WHEN("Constructed with default number of fields and chunk size.") {
      CHECK_NOTHROW(AlignmentReader::FromFile(input_file));
      /*AlignmentReader reader{AlignmentReader::FromFile(input_file)};

      THEN("Field number and chunk size are at default values.") {
        CHECK(reader.NumFields() == 12);
        CHECK(reader.ChunkSize() == 128 * 1000 * 1000);
      }*/
    }
  }
}

} // namespace

} // namespace test

} // namespace paste_alignments

when I uncomment the commented part, the exit code changes to 2 and ctest still reports success.

Here is what happens when I run ctest (both with and without the commented portion):

$ ctest
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
    Start 1: alignment_reader_test
1/1 Test #1: alignment_reader_test ............   Passed    0.16 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.17 sec

Here is what I get if I run the test individually and check exit status (with the stuff commented out):

$ ./test/alignment_reader_test 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
alignment_reader_test is a Catch v2.12.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: Test correctness of AlignmentReader::FromFile.
      Given: The name of a valid input file.
       When: Constructed with default number of fields and chunk size.
-------------------------------------------------------------------------------
/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:64
...............................................................................

/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:65: FAILED:
  CHECK_NOTHROW( AlignmentReader::FromFile(input_file) )
due to unexpected exception with message:
  Unable to open file: 'test_data/valid_alignment_file.tsv'.

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

$ echo $?
1

The only things that change when I include the commented portion is that two tests fail and the exit status is 2 instead of 1.

Here with --verbose flag:

$ ctest --verbose
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: alignment_reader_test

1: Test command: /Users/Jasper/cpp_projects/PasteAlignments/debug/test/alignment_reader_test
1: Test timeout computed to be: 1500
1: ===============================================================================
1: All tests passed (1 assertion in 1 test case)
1: 
1/1 Test #1: alignment_reader_test ............   Passed    0.36 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.36 sec

I didn't change the ctest configurations (as suggested here). CMakeLists.txt looks like this:

add_executable(alignment_reader_test
        "${PROJECT_SOURCE_DIR}/test/alignment_reader_test.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_reader.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_batch.cc"
        "${PROJECT_SOURCE_DIR}/src/scoring_system.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment.cc"
        "${PROJECT_SOURCE_DIR}/src/helpers.cc")
target_include_directories(alignment_reader_test PUBLIC
        "${PROJECT_SOURCE_DIR}/test"
        "${PROJECT_SOURCE_DIR}/include"
        "${PROJECT_SOURCE_DIR}/lib/catch/include")
add_test(NAME alignment_reader_test COMMAND alignment_reader_test)

Answer 1

I found the issue thanks to @Tsyvarev.

The build directory structure is as follows:

build/
|--test/
   |--test_executable
   |--test_data/
      |--test_datafile

When CMake runs test_executable it runs it from within the directory test.

When I ran the executable separately, I ran it from the build directory. (./test/test_executable).

Inside the unit-test code, the test_datafile is referred to as test_data/test_datafile. This is not recognized when the test is run from the build executable (as I did) as opposed to when it's run from the build/test directory.

Therefore, when ctest ran the test, it actually succeeded as it should.

Indeed, if I cd into test first, the test has exit code 0, as it should.