Add miraculix testing library

[m-bock]

Hi, first of all thanks for writing purenix! I think this is a really useful and promising approach to write typed nix!
It inspired me to write this small testing library which is inspired by tasty. With this PR I'd like to ask if it could be listed in the package set.

Working with the current purenix version was really smooth.
The only issue I encountered was the quoting thing already mentioned in here.

[cdepillabout]

Wow, this is really nice! This isn't just a nice PureNix library, this would just be a convenient library in general for Nix.

I almost feel like you should wrap this up in a thin Nix API and release it as a normal Nix library. For instance, here's the tests function from the example in your README:

{
  tests =
    module."Test.Miraculix.TestTree".testGroup
      "Lib functions"
      [ ( module."Test.Miraculix.TestTree".testGroup
            "Math"
            [ ( module."Data.Function".apply
                  ( module."Test.Miraculix.TestTree".testCase "addition" )
                  ( module."Test.Miraculix.Assertion".assertEq module."Data.Show".showInt module."Data.Eq".eqInt (module."Data.Semiring".add module."Data.Semiring".semiringInt 1 1) 2 )
              )
              ( module."Data.Function".apply
                  ( module."Test.Miraculix.TestTree".testCase "muliplication")
                  ( module."Test.Miraculix.Assertion".assertEq module."Data.Show".showInt module."Data.Eq".eqInt (module."Data.Semiring".mul module."Data.Semiring".semiringInt 3 4) 11 )
              )
            ]
        )
        ( module."Test.Miraculix.TestTree".testGroup "Strings"
            [ ( module."Data.Function".apply
                  ( module."Test.Miraculix.TestTree".testCase "sorts a list of numbers" )
                  ( module."Test.Miraculix.Assertion".assertEq
                      ( module."Data.Show".showArray module."Data.Show".showInt )
                      ( module."Data.Eq".eqArray module."Data.Eq".eqInt )
                      ( module."Data.Array".sort module."Data.Ord".ordInt [2 3 1] )
                      [1 2 3]
                  )
              )
              ( module."Data.Function".apply
                  ( module."Test.Miraculix.TestTree".testCase "sorts a list of characters" )
                  ( module."Test.Miraculix.Assertion".assertEq
                      ( module."Data.Show".showArray module."Data.Show".showChar )
                      ( module."Data.Eq".eqArray module."Data.Eq".eqChar )
                      ( module."Data.Array".sort module."Data.Ord".ordChar ["c" "b" "a"] )
                      ["a" "b"]
                  )
              )
            ]
        )
      ];
}

This is both easy to read and write. But if you clean up some of the unnecessary PureNix stuff, it looks even better:

{
  tests =
    testGroup
      "Lib functions"
      [ (testGroup
          "Math"
          [ (testCase "addition" (assertEq showInt eqInt (add semiringInt 1 1) 2))
            (testCase "muliplication" (assertEq showInt eqInt (mul semiringInt 3 4) 11))
          ]
        )
        (testGroup
          "Strings"
          [ (testCase
              "sorts a list of numbers"
              (assertEq (showArray showInt) (eqArray eqInt) (sort ordInt [2 3 1]) [1 2 3])
            )
            (testCase
              "sorts a list of characters"
              (assertEq (showArray showChar) (eqArray eqChar) (sort ordChar ["c" "b" "a"]) ["a" "b"])
            )
          ]
        )
      ];
}

Aside from the typeclass instances (like showArray, showChar, ordChar, eqInt, etc), this looks really nice.

[cdepillabout]

Also, I see in the most recent version of miraculix, you're using purescript-effect: https://github.com/thought2/purescript-effect

I wrote a little about purescript-effect in this issue: purenix-org/purenix#37. Basically, since Nix is pure, I didn't see a good reason to port purescript-effect. Although it's API is pretty simple, so it would of course be possible to port for compatibility reasons.

It looks like you're using Effect both for logging and abort.

The two concerns I would have about this are:

• In Haskell, you can have putStrLn :: String -> IO (), and you can be relatively (completely?) sure that the string you pass in will actually get printed. But in Nix, all you have is builtins.trace, which is more like String -> a -> a. You don't really have any guarantee that Nix will print what you want it to print. Although, now that I think about it, maybe wrapping all calls to trace in Effect and only running Effect at the top of your program, you do have a more assurance that the String you pass to trace will actually be printed?

• In Haskell/PureScript, it is convenient to wrap up calls to things like abort in Effect because you have a way of catching the abortion. So you'd have a function like catch :: IO a -> (Exception -> IO a) -> IO a. But in Nix, I don't think there is any way of catching abort failures:

  nix-repl> builtins.tryEval (builtins.abort "hello")
  error: evaluation aborted with the following error message: 'hello'

  Without a way to write catch, I didn't think that having abort be in Effect would be that helpful.

  Although now taking a closer look at tryEval, it does look like assert failures and throw can be caught (but you unfortunately can't get the value of the argument passed to throw):

  nix-repl> builtins.tryEval (assert 1 == 2; "hello") 
  { success = false; value = false; }
  nix-repl> builtins.tryEval (builtins.throw "hello")
  { success = false; value = false; }

So yeah, after thinking this through, it seems like I may just have been wrong here and purescript-effect would be useful.

If you want to move forward with purescript-effect, ping me and we can transfer it to purenix-org and I'll make you a maintainer there.

[cdepillabout]

This is just another thought I had (but I don't think you should necessarily let it determine the direction you take miraculix).

I'd like to get support in PureNix for handling spago test (purenix-org/purenix#39). After that is in place, it would be great to add tests to some of the packages in the standard library (the purescript libraries in https://github.com/purenix-org). It would of course be nice to have a testing framework to use for this.

The problem is that the more dependencies the testing framework has, the fewer packages in the standard library that could use the testing framework. For instance, if the testing framework doesn't have any dependencies, it could be used to test every package in the standard library. Oppositely, if the testing framework has a dependency on a package like purescript-arrays, then it likely couldn't be used to test any of the packages currently available (since purescript-arrays has almost every package in its transitive dependencies).

Moving forward, it might be ideal to have two separate testing frameworks. One like miraculix that has as many dependencies as necessary and provides lots of features out of the box, and a bare-bones testing framework that has no (or minimal) dependencies. End-users would generally use miraculix, while the standard-library packages would use the bare-bones testing framework.

[m-bock]

I agree, it's quite debatable if an Effect type does make any sense in a purenix context. When I first started to think about this I was almost sure that having Effect would not be necessary because of Nix's already pure nature.
But after a while I started noticing some cases where it may still make sense.

As you have seen, miraculix 0.1.0 does not use Effect. So what motivated me for the refactoring using Effect in 0.2.0? Correct logging in the context of a test runner is an important part of its behavior. It's more than sheer logging for debugging purposes. So I wanted the types to reflect which functions do log and which don't. Moreover, it looked really unnatural to me to work with trace all the time and I got easily confused about the order of execution. So I preferred expressing the logging in a "sequential" do notation. the first issue I encountered was the exact thing that you mention. I could not guarantee that every trace :: String -> Effect Unit was actually executed. In order to make this work, I ended up defining bindE in the effect library in a strict way like this: https://github.com/thought2/purescript-effect/blob/main/src/Effect.nix#L4

At the moment I can only say that this works out. But I'm not sure if this implementation may have other unwanted implications. So from a practical point of view I'd say I already gained from using Effect for logging. If for some reason it will turn out that using Effect in purenix is an anti pattern, I could still use some unsafePerformEffect to avoid exposing the type in the API.

Regarding the abort, here's an excerpt from the Nix builtins docs: "tryEval will only prevent errors created by throw or assert from being thrown. Errors tryEval will not catch are for example those created by abort and type errors generated by builtins."

And as you say, unfortunately there's no way to catch the thrown value. But with those limitations in mind, a catch :: Effect a -> Effect a -> Effect a could make sense.

If higher level libraries make sure that normal functions don't throw, then an intentional possible throw could be reflected in the Effect type.

However, I don't think that Nix functions like fetchFromGithub should be wrapped in Effect. I think it's a pretty clear case if you have to specify a sha to validate the return type. Then the operation is absolutely predictable.

But there are also functions like readFile or getEnv which will influence the output hash of the derivation depending on the file content or set environment variables. In this sense Nix is not 100% pure. Checking if an expression is deterministic can be done by nix-build and then nix-build --check. The second command will tell you if rebuilding has produced another output hash. This can be easily tested if you use $RANDOM inside a writeText context.

At the moment I think: If those functions would be typed as e.g. getEnv :: Effect String it should be possible to state that a purenix program not written in Effect would be fully deterministic. But there may be some things I'm overlooking here. But on the other hand, you can also have this guarantee with the --restrict-eval flag.

I had the idea of having a standard library for purenix. It could provide a set of well typed bindings to the most important builtin functions. Especially the very nix specific ones which will not appear in the data type oriented packages (e.g. arrays, ...)
Things like fetchFromGit readFile, abort, throw, even something like import. And maybe also come up with a type concept for store paths, e.g. the return value of fetchFromGit.
In such a project those questions about the Effect type become very interesting.

Regarding the idea of having an additional dependency free testing framework: I think this could really make sense. In the end it would mean that some portions of basic libraries would have to be copy/pasted into the test framework. But this would bring in a clean dependency graph. I can very well imagine implementing this in the future.

And yes, maybe I'll consider publishing the lib as a nix package, too.

Thanks a lot for the suggestions! I'm excited to see where the discussion about the Effect will go. If there's clarity about what makes most sense, having a standard lib could really give a very nice out-of the box experience when using purenix. Looking forward to continue working on this. If it turns out that an effect library is useful, we can transfer it to purenix-org and I'll be glad to maintain it further.