Fatal exceptions in tests are ignored

[Arthurm1]

Describe the bug

Create the following munit test...

class IllegalAccess extends munit.FunSuite {
    test("Handle IllegalAccessError") {
        throw new IllegalAccessError("This should cause test to fail")
    }
}

Run it under Metals.

Result is...

IllegalAccess:
Execution took 0ms
No test suite was run

Instead of reporting a failed test, the test is ignored and the suite completes semi-successfully. It also causes any other tests in the suite to be ignored so it's hard to realise that the tests are failing.

This exception is thrown if --add-exports is missing from code that requires it.

Here you can see that IllegalAccessError is a subclass of java.lang.LinkageError.
Here you can see that java.lang.LinkageError is treated as fatal.
Here, Munit only handles NonFatal errors.
Here JUnit runner does seem to report the error.

I'd add a stacktrace but there isn't one printed as it silently fails.

I'm not sure if this is an issue with Bloop, Metals, MUnit, JUnit or the way Scala classifies NonFatal errors.

Expected behavior

I think this type of error should be reported and the test should gracefully fail. It's not like we're out of memory.

Operating system

Windows

Editor/Extension

VS Code

Version of Metals

v0.11.12

Extra context or search terms

No response

[filipwiech]

Could this be connected with scalameta/munit#650? 🤔

[tgodzik]

Thanks for reporting! I think the issue might lie with munit. Not sure where the tests are invoked, but I would bet that is the reason, since we don't get a failure to report

[Arthurm1]

This was fixed in scalameta/munit#669 and released in v1.0.0-M9 and Metals is now on v1.0.0-M11

Quick test shows it works in Metals.