base library

[eed3si9n]

• MiniTest or uTest are current front runners
• Use JUnit? (what about Scala Native?)
• and testz?

criteria

• signs Scala CLA (transfers copyright to Lightbend/EPFL)
• supports JVM, Scala.JS, and Scala Native
• (therefore) supports Scala 2.11, 2.12, and 2.13
• integrates with sbt's sbt/test-interface
• other than that, smaller the better

[eed3si9n]

In terms of simplicity and ready sbt integration, I think Minitest is the best candidate for the starting point https://github.com/monix/minitest.

import minitest._

object MySimpleSuite extends SimpleTestSuite {
  test("should be") {
    assertEquals(2, 1 + 1)
  }

  test("should not be") {
    assert(1 + 1 != 3)
  }

  test("should throw") {
    class DummyException extends RuntimeException("DUMMY")
    def test(): String = throw new DummyException

    intercept[DummyException] {
      test()
    }
  }

  test("test result of") {
    assertResult("hello world") {
      "hello" + " " + "world"
    }
  }
}

[martijnhoekstra]

I made a list of features/properties that test frameworks could have, that may be relevant for consideration. I believe all properties to be useful for a minimal test framework, though none (other than plain assertions) are probably entirely essential. Comparing the libraries mentioned abovem minitest, utest and testz on these points could be a nice start. Pinging framework authors @edmundnoble @lihaoyi-databricks and @alexandru

Not mentioned were scala test-state, ffstest, scalacheck, scalaprops, discipline, claimant and hedgehog, though I suppose though could be evaluated on the same points.

• Size (# jars, binary size, loc)
• Plain assertion
• Equals assertion
• Testing in/with/of Future
• Testing in/with/of other effects
• Testing Exceptions
• Other assertion/matchers
• Plain assertion failure messages
• Equals assertion failure messages (diffs?)
• Running test(suite)s stand alone
• Running test(suite)s in sbt
• License

[martijnhoekstra]

Starting with what I see from minitest above:

• size: 1 jar, 117 kb, ?? loc
• plain assertion: test("should not be") { assert(1 + 1 != 3) }
• equals assertion: test("should be") { assertEquals(foo, bar) }
• testing Futures: ???
• testing other effects: ???
• exceptions:

def test(): String = throw new DummyException
intercept[DummyException] { test() }

• other assertion/matchers: ???
• plain assertion failure messages: ???
• equals assertion failure messages: ???
• running stand-alone: ???
• running in sbt: ???
• license: apache 2

[alexandru]

Hello,

In case you want to use Minitest or parts of it, I agree with transferring the copyright to Lightbend or whatever license changes you want.

For the criteria ...

Testing Future is supported via testAsync:

testAsync("some future") {
  Future {
    assertEquals(foo, bar)
  }
}

Testing "other effects" is not supported directly, since that would imply extra dependencies, however all you really need is Future support, since all others can be translated to Future. You can for example have something like this in your project and all is good:

import cats.effect._

def testEffect[F[_] : Effect](name: String)(f: => F[Unit]): Unit =
  testAsync(name) {
    val p = Promise[Unit]()
    Effect[F].runAsync(f)(r => p.complete(r.toTry))
    p.future
  }

Could use some improvements ... assertEquals for example could do a smart diff between the tested data structures. I've seen other libraries do this and seems to be useful.

running stand-alone

Minitest does not run standalone, depending entirely on sbt. It could be made to run standalone, but that's more work than I wanted to do and I never saw the need.

[eed3si9n]

@alexandru Thanks for your generous offer. I've pushed the commits from master into this repo :)

[martijnhoekstra]

So I take it the discussion is effectively over then?

[eed3si9n]

@martijnhoekstra Yea. We're moving forward with using Minitest as the starting point.

[martijnhoekstra]

So whats the process from here for competing proposals?

[eed3si9n]

@martijnhoekstra I guess it depends on what you mean by competing proposals. This is a result reached from a long-winding discussion that started in scala/scala-dev#641 and continued over here, and I for one is ready to move on to looking at code, and focusing on details.

[martijnhoekstra]

Sure, this is your proposal, your call, so counter proposals should probably live in a separate repository. So that's what I'm asking about - at the point where a decision has to be made which strawman to choose, what is the process that chooses that going to look like?

…

On Thu, Jul 25, 2019, 21:21 eugene yokota ***@***.***> wrote: @martijnhoekstra <https://github.com/martijnhoekstra> I guess it depends on what you mean by competing proposals. This is a result reached from a long-winding discussion that started in scala/scala-dev#641 <scala/scala-dev#641> and continued over here, and I for one is ready to move on to looking at code, and focusing on details. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXOEIYQWDWBEVDXBCERDDQBH4NDANCNFSM4IECSORQ> .

[eed3si9n]

So that's what I'm asking about - at the point where a decision has to be
made which strawman to choose, what is the process that chooses that going
to look like?

I have not thought about the process. It's supposed to a summer project, so at the end of summer we'd both present modules and/or pull request to scala/scala, and people can comment on it?

[martijnhoekstra]

We seem to be having the same question about that.

…

On Thu, Jul 25, 2019, 22:19 eugene yokota ***@***.***> wrote: So that's what I'm asking about - at the point where a decision has to be made which strawman to choose, what is the process that chooses that going to look like? I have not thought about the process. It's supposed to a summer project <https://contributors.scala-lang.org/t/summer-of-usability/3484>, so at the end of summer we'd both present modules and/or pull request to scala/scala, and people can comment on it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXOEK75JTZAJGYK3IOVEDQBIDFDANCNFSM4IECSORQ> .