SBT test does not run zio tests despite inclusion of test framework ZTestFramework

Question

Edited following comments by @GastónSchabas to include a minimal reproducible example.

I have a project in Scala 2.3.11, sbt 1.9.3, zio 2.0.15, zio-test 2.0.15.

In the project, I have a mix of scalatest tests org.scalatest.flatspec.AnyFlatSpec and ZIO Junit tests zio.test.junit.JUnitRunnableSpec.

Here is the build.sbt file:

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.11"

val zioVersion = "2.0.15"
val scalaTestVersion = "3.2.16"

libraryDependencies ++= Seq(
  "org.scalactic"     %% "scalactic"         % scalaTestVersion,
  "org.scalatest"     %% "scalatest"         % scalaTestVersion % Test,
  "dev.zio"           %% "zio"               % zioVersion,
  "dev.zio"           %% "zio-test"          % zioVersion % Test,
  "dev.zio"           %% "zio-test-sbt"      % zioVersion % Test,
  "dev.zio"           %% "zio-test-magnolia" % zioVersion % Test,
  "dev.zio"           %% "zio-test-junit"    % zioVersion % Test
)

testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")

Here is the build.properties file:

sbt.version = 1.9.3

In src/test, I include exactly two tests, one scalatest and one ziotest, as follows:

import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class ScalaTest1 extends AnyFlatSpec with Matchers {
  "test" should "succeed" in {
    42 shouldEqual 42
  }
}

import zio._
import zio.test.junit.JUnitRunnableSpec
import zio.test.{Spec, TestEnvironment, assertTrue}

class ZioTest1 extends JUnitRunnableSpec {
  override def spec = suite("ZioTest1")(
    test("test1") {
      for {
        number <- ZIO.succeed(42)
      } yield {
        assertTrue(number == 42)
      }
    }
  )
}

My directory structure looks like this:

- src
  - test
    - scala
      ScalaTest1.scala
      ZioTest1.scala
- project
  build.properties
build.sbt

In SBT, if I try:

% sbt
 % sbt
[info] welcome to sbt 1.9.3 (Eclipse Adoptium Java 17.0.2)
...
sbt:zio-test-with-sbt> show testFrameworks
[info] * TestFramework(org.scalacheck.ScalaCheckFramework)
[info] * TestFramework(org.specs2.runner.Specs2Framework, org.specs2.runner.SpecsFramework)
[info] * TestFramework(org.specs.runner.SpecsFramework)
[info] * TestFramework(org.scalatest.tools.Framework, org.scalatest.tools.ScalaTestFramework)
[info] * TestFramework(com.novocode.junit.JUnitFramework)
[info] * TestFramework(munit.Framework)
[info] * TestFramework(zio.test.sbt.ZTestFramework)
[info] * TestFramework(weaver.framework.CatsEffect)
[info] * TestFramework(hedgehog.sbt.Framework)
[info] * TestFramework(zio.test.sbt.ZTestFramework)
sbt:zio-test-with-sbt> 

So, ZTestFramework has been correctly added.

I can run the ZIO Junit tests easily from within Intellij.

However, when I ran sbt test, I was expecting my ZIO tests to run. Instead, only the scalatest tests ran.

 % sbt test
[info] welcome to sbt 1.9.3 (Eclipse Adoptium Java 17.0.2)
...
[info] ScalaTest1:
[info] test
[info] - should succeed
[info] Run completed in 274 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 1 s, completed Aug 9, 2023, 4:04:43 PM

Any idea why it's not picking up my zio tests? Is there any way to debug the ZTestFramework further?

Answer 1

The short answer is, ZioTest1 have to be changed from class to object. As it is detailed in Writing Our First ZIO Test

NOTE ⓘ

In order to have runnable tests, the ZIOSpecDefault trait must be extended by an object that implements the spec method. If we extend this trait in a class, the test runner will not be able to find the tests.

Replacing

class ZioTest1 extends JUnitRunnableSpec {
  // ...
}

with

object ZioTest1 extends JUnitRunnableSpec {
  // ...
}

Should make your test start running with sbt test.

I think you copy pasted a test from the ScalaTest ones and you changed the traits mixed to the class, but forgot to make it an object. The same problem happens when you make an object a test from ScalaTest. In the previous example I provided, you can see that difference

---

Creating the following files, I was able to run all the tests with the command sbt test.

• build.sbt

libraryDependencies ++= Seq(
  "org.scalactic"     %% "scalactic"         % "3.2.16",
  "org.scalatest"     %% "scalatest"         % "3.2.16"   % Test,
  "org.scalatestplus" %% "scalacheck-1-17"   % "3.2.16.0" % Test,
  "dev.zio"           %% "zio"               % zioVersion,
  "dev.zio"           %% "zio-test"          % zioVersion % Test,
  "dev.zio"           %% "zio-test-sbt"      % zioVersion % Test,
  "dev.zio"           %% "zio-test-magnolia" % zioVersion % Test,
  "dev.zio"           %% "zio-test-junit"    % zioVersion % Test
),
testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")

• ZioHelloWorld.scala

import zio._
import java.io.IOException

object ZioHelloWorld {
  def sayHello: ZIO[Any, IOException, Unit] =
    Console.printLine("Hello, World!")
}

• ZioHelloWorldSpec.scala

import ZioHelloWorld._
import zio.test._

object ZioHelloWorldSpec extends ZIOSpecDefault {
  def spec =
    suite("ZioHelloWorldSpec")(test("sayHello correctly displays output zio test") {
      for {
        _ <- sayHello
        output <- TestConsole.output
      } yield assertTrue(output == Vector("Hello, World!\n"))
    })
}

• ZioJunitHelloWorldSpec.scala

you have two ways of Integrating ZIO Test with JUnit. One is as the example added below, the other one is adding the annotation @RunWith(classOf[ZTestJUnitRunner]) above the class definition, mix ZIOSpecDefault instead of JUnitRunnableSpec and also add the library "com.github.sbt" % "junit-interface" % "0.13.3" % Test

import ZioHelloWorld._
import zio.test._
import zio.test.junit.JUnitRunnableSpec

object ZioJunitHelloWorldSpec extends JUnitRunnableSpec {
  def spec =
    suite("ZioJunitHelloWorldSpec")(test("sayHello correctly displays output using junit") {
      for {
        _ <- sayHello
        output <- TestConsole.output
      } yield assertTrue(output == Vector("Hello, World!\n"))
    })
}

• HelloScalaTest.scala

import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.matchers.should.Matchers
import org.scalatestplus.scalacheck.ScalaCheckPropertyChecks

class HelloScalaTest extends AnyFunSuite with Matchers with ScalaCheckPropertyChecks {

  test("1 should be 1") {
    1 should be(1)
  }

  test("(a + b) should be (b + a)") {
    forAll { (a: Int, b: Int) =>
      (a + b) should be(b + a)
    }
  }

}

Once I executed sbt test I got the following output

+ ZioHelloWorldSpec
Hello, World!
Hello, World!
  + sayHello correctly displays output zio test
+ ZioJunitHelloWorldSpec
  + sayHello correctly displays output using junit
[info] HelloScalaTest:
[info] - 1 should be 1
[info] - (a + b) should be (b + a)
4 tests passed. 0 tests failed. 0 tests ignored.