How can I intercept perl Test::More's BAIL_OUT() and continue the next test?

Question

Perl's Test::More doesn't seem to offer built-in way to say "bail out of just this test script, and continue with the next one".

You can exit() or die() but it doesn't give you very useful TAP output or prove output, just something like

t/foo.pl       (Wstat: 256 Tests: 5 Failed: 0)
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 38 tests but ran 5.

which isn't super helpful.

If you BAIL_OUT("reason") you get more useful output, but all tests get aborted, not just the current test script.

Is there a sensible way to print a diagnostic message that'll be seen on prove's stderr and in the summary output when the harness exits, then jump to the next test script?

Answer 1

I grovelled through more of the Perl standard library than I ever wanted to to come up with this as my best option so far:

# Install a filter to prevent BAIL_OUT() from aborting the whole
# run.
Test2::API::test2_add_callback_context_init(sub {
    my $context = shift;
    $context->hub->filter(sub {
        my ($hub, $event) = @_;

        #
        # If you want to see details on the event contents use:
        #
        #print("EVENT FILTER: " . Dumper($event) . "\n");

        # Turn BAIL_OUT() into die(), or maybe $ctx->throw(). We can't easily
        # get the context.
        if ($event->isa('Test2::Event::Bail')) {

            # Ideally we'd produce a Test2::Event with a Test2::EventFacet::Control
            # with terminate true, and a Test2::EventFacet::Trace . For for
            # now we'll just log the BAIL_OUT() event and exit.
            #
            # If we ->throw it instead, we get a big stack for little benefit.
            # And END { } blocks will still get run when we exit(...) here so
            # appropriate cleanup still happens.
            #
            $event->trace->alert($event->reason);
            exit(1);
        }
        return $event;
    });
});

This will cause a test script that calls BAIL_OUT("FOOBAR"); to bail with something like

FOOBAR at t/foo.pl line 4.
# Looks like your test exited with 1 just after 5.

then prove stderr will show:

t/foo.pl ......... 1/38 
# Looks like your test exited with 1 just after 5.

and the summary will show

t/foo.pl       (Wstat: 256 Tests: 5 Failed: 0)
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 38 tests but ran 5.

This is far from ideal, as the stderr and summary doesn't tell us what went wrong.

But it prevents the BAIL_OUT() from nuking the rest of the run.

One downside is that this won't help provide more info for tests that die(). So I'm still looking for better options.

Answer 2

Many times I just wrap my tests inside subtest 'name' => sub { ... } and use return inside like the following:

use Test::More;

subtest 'part 1' => sub {

    plan tests => 10;

    lives_ok {
        ...
    } "this went smooth"

    or return; # no use of further testing

    # more tests

    fail "I'm done here, bail out!" or return

    # more tests

};

done_testing;

This works, because passing tests return 'true' and failing test return 'false' and thus the 2nd part of the or gets evaluated.

This won't break TAP, you can 'bail out' of subtest, or the entire test (if wrapped) and keep running the rest of the tests.

My use case is that if I can not instantiate a test-object to be further tested, the remaining tests in that subtest are just useless, non of those test would make sense, and I only get cluttered test reports.

You could potentially do:

    lives_ok {
        ...
    } "Yay"

    or fail "No use to continue here, bye!" or return;

to be more explicit that you do exit there.