Anyway to get TestNG to not run a suite if a class method does not exist?

Question

TestNG (6.14.3) for some reason will continue to run if one of the class methods doesn't exist. What it seems to do is just execute the next @Test method in the class file. Is there any way to get TestNG to fail the suite if one of the class test methods does not exist? In the following suite file, even though one of the test methods does not exist in the class file, it will just pick up what seems to be a random @test method and continue on it's merry way:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="testSuite" preserve-order="true">
    <parameter name="sProject" value="mine"/>
    <listeners>
        <listener class-name="framework.Listener"/>
    </listeners>
    <test name="TestSuite-Part1" preserve-order="true">
        <classes>
            <class name="tests.HomePageTest">
                <methods>
                    <include name="gotoHomePageTest"/>
                </methods>
            </class>
            <class name="tests.MiniBannerCardTest">
                <methods>
                    <include name="thisMethodDoesNotExist"/>
                    <include name="verifyDeliveryZipCodeSetTest"/>
                </methods>
            </class>
            <class name="tests.HeaderCardTest">
                <methods>
                    <include name="clickSignInOrRegisterLinkTest"/>
                </methods>
            </class>
        </classes>
    </test>
</suite>

I've tried the following in the onStart in a suite listener which works great on the dev machine! However, there is no ./src/main/java nor any source files on the runner VM for it to process, so it just aborts saying the path is non-existent. Any ideas?

    Log.info("**********About to begin executing suite: " + suite.getName());
    // getting included method names in XML suite file
    List<String> inclduedMethodsInXmlSuite =
            suite.getXmlSuite().getTests().stream()
                  .flatMap(xmlTest -> xmlTest.getClasses().stream().flatMap(xmlClass -> xmlClass.getIncludedMethods().stream().map(XmlInclude::getName)))
                  .collect(Collectors.toList());
    // getting all test methods
    List<String> testMethods = new ArrayList<>();
    try (Stream<Path> paths = Files.walk(Paths.get("./src/main/java")))
    { // path to test classes directory
        testMethods = paths
                .filter(path -> path.getFileName().toString().endsWith(".java")) // filtering only classes, not directories
                .map(path -> path.getParent().getFileName().toString() + "." + path.getFileName().toString()) // mapping to qualified name, e.g test.TestClass
                .map(file -> file.substring(0, file.length() - 5)) // removing ".java"
                .map(clazz ->
                     {
                         try
                         {
                             return Class.forName(clazz); // casting to Class object
                         }
                         catch (ClassNotFoundException e)
                         {
                             e.printStackTrace();
                             return null;
                         }
                     }
                    )
                .flatMap(clazz -> Arrays.stream(clazz.getDeclaredMethods()).filter(method -> method.isAnnotationPresent(Test.class)))
                .map(Method::getName) // mapping to method name
                .collect(Collectors.toList());
    }
    catch (IOException e)
    {
        e.printStackTrace();
    }
    // checking if any of xml suite methods does not exist in test methods
    final List<String> finalTestMethods = testMethods;
    List<String> notSupprtedMethods = inclduedMethodsInXmlSuite.stream()
                                                               .filter(xmlMethod -> !finalTestMethods.contains(xmlMethod))
                                                               .collect(Collectors.toList());
    if (notSupprtedMethods.size() > 0)
    {
        System.out.println("*****=====Unsupported methods in XML Suite file=====*****:");
        notSupprtedMethods.forEach(System.out::println);
        System.exit(0);
    }

EDIT: using 7.4.0 it seems to just skip the missing method rather than run the next one. No log or any indication it had skipped though.

Answer 1

TestNG doesn't check if the method mentioned in <include> is available or not. It is actually the other way. i.e. It first picks all the test methods in the specified <class>, then for each method it checks whether it is included or excluded in the xml and then execution is carried out based on this data. It does not mark the method as skipped because it is not configured to run.

But from your comment, I could understand that you are good if there is some message logged. TestNG does log such exclusions.

So when any method is not included in <include> tag, it logs the message at INFO level. For that, you need to add the below log level in your log4testng.properties file.

log4testng.logger.org.testng.internal.Utils=INFO

Assume that you have firstTest method in your test class (package a.b.c) and you accidently specified the name as firsTest in the xml, then after adding the log properties, the below message would be available.

[Utils] [INFO] [XmlMethodSelector] Excluding method a.b.c.firstTest()

Also make sure the verbose is set to 4 in <test>

<test name="TestSuite-Part1" preserve-order="true" verbose="4">