handle tests using TAP::Harness, how to print output when test exits

Question

I am trying the whole day to find out the answer, but I didn't find anything. I wrote some tests using test::more (test1.t, test2.t, test3.t ...). and I wrote a main perl script (main.pl) that handles all the tests using TAP::Harness and print the output in a JUnit format using formatter_class => 'TAP::Formatter::JUnit. In my tests I use the BAIL_OUT function. The problem is that when a test is bailed out, the main script also exits and there is no output at all. If, for example test3.t bailed_out, I need to see the results for test1.t and test2.t. how can I do that?

I can't use exit or die instead of BAIL_OUT because I don't want the other tests to continue. (If test3.t was BAIL_OUT I don't want that test4.t will run.)

can someone please help me? I need to see the results for the tests that were running before the bailed out test.

Thanks.

Answer 1

According to the Test::More docs:

BAIL_OUT($reason);

Indicates to the harness that things are going so badly all testing should terminate.
This includes the running of any additional test scripts.

So that explains why your suite aborts.

You might want to consider die_on_fail from Test::Most, or skip_all, depending on the reason for the BAIL_OUT.

EDIT: Looks like Test::Builder doesn't have any intention of printing out a summary when it exits on "catastrophic failure" according to the source code:

sub BAIL_OUT {
    my( $self, $reason ) = @_;

    $self->{Bailed_Out} = 1;
    $self->_print("Bail out!  $reason");
    exit 255;
}

# Don't do an ending if we bailed out.
if( $self->{Bailed_Out} ) {
    $self->is_passing(0);
    return;
}

However, that Bailed_Out flag is only ever used to consider printing out summary diagnostics, and since Test::More exposes the underlying Test::Builder object, you can probably just tweak the BAIL_OUT subroutine and no set this flag. All untested, of course; YMMV.

Answer 2

Instead of passing in all tests to one TAP::Harness, you need to pass in one test at a time to the Harness in case of a BAIL_OUT

I haven't seen your code, so here is a sample of what I mean. Adjust to include the formatter and whatever else you need.

use TAP::Harness;

my $harness = TAP::Harness->new({ merge => 0 });
my $tests = ['t/test1.t', 't/test2.t'];

foreach my $test (@$tests) {
    eval {
        $harness->runtests([$test]);
    }; if ($@) {
        # create new harness object if the previous fails catastrophically.
        $harness = TAP::Harness->new({ merge => 0 });
    }
}