I am building Apache Spark on a Z mainframe. I'm using Apache Maven for the build/test. Spark has a ton of tests that run, but due to some resource restrictions for my user on the system, I need to be able to run some test Suites by themselves. In order to do this however, maven wants to recompile everything, both production code and tests. Is there anyway to just re-run tests without having to recompile the world? Some project depend on each other, and when I need to run these 1 test, it takes over an hour to run the single test.
The process I take to build and test are:
Build Spark (compile and package without running tests); package it using the make-distribution.sh script.
mvn -e -Phive -Phive-thriftserver -Phadoop-3 -Puser-defined-protoc -DskipDefaultProtoc -Dmaven.test.skip=false -DskipTests clean package
Then run tests.
mvn -q -e -fn -Phive -Phive-thriftserver -Phadoop-3 -Puser-defined-protoc -DskipDefaultProtoc test
Then re-run tests that are missing from the list.
# Run tests that exist but didn't run due to JVM failure during initial run.
mvn -pl connector/connect/client/jvm -am -q -e -fn -Phive -Phive-thriftserver -Phadoop-3 -Puser-defined-protoc -DskipDefaultProtoc -DwildcardSuites=org.apache.spark.sql.test.ConnectFunSuite -Dtest=none test
Then re-run tests that have failed. (possibly flaky tests)
mvn -pl mllib -am -q -e -fn -Phive -Phive-thriftserver -Phadoop-3 -Puser-defined-protoc -DskipDefaultProtoc -DwildcardSuites=org.apache.spark.ml.MLEventsSuite -Dtest=none test
It is worth noting that the -am is required for Spark projects because I keep hitting a problem by the time it reaches the expected module, that it doesn't have the necessary classes loaded and will error if I don't use -am. If there is a configuration that is needed to pull in already compiled classes before a given project, that'd be helpful too.
This process takes 16+ hours to complete a single Jenkins build. If possible, I'd like to build and package. (compile all source files including test) Then just run the tests as needed.
--- Update ----
I did try the surefire:test plugin directly, and it does save me from having to recompile the tests; but I still run into an issue with not picking up previously compiled source code, such as missing from CLASSPATH. So my tests fail due to unable to find a given package/function as it doesn't exist. Also it appears to only run Java tests and not Scala tests.