Include tests loading failures in report

[goldbergyoni]

Hi,

I ran few test suites, most failed to run due to compilation error (test didnt actually ran). One suite ran successfully with 2 tests that passed.

The output .xml file reports 2 tests where both succeeded. Doesn't it make sense to report all the failing tests in the tests suites as failures (Even though they didn't run, they failed to compile)?

Yoni

[palmerj3]

@i0natan thanks for submitting an issue!

You do bring up a good point and this is something I would love to address.
The jUnit spec offers us three types of test results (succeed, fail, error). In your case the final result should have, ideally, been 2 succeed and n-error.

Jest, however, doesn't give any information to a testResultsProcessor for the error cases.

You can see here where I've left a comment kind of vaguely suggesting this :)
At the very least I think what your issue highlights is that I should document this.

https://github.com/palmerj3/jest-junit/blob/master/utils/buildJsonResults.js#L39

[palmerj3]

I'm going to leave this issue open and spend a bit more time re-investigating this. It's a feature I would absolutely LOVE to add if/when it is possible.

[dangoslen]

Hi! New to open source, so hopefully not out of place joining the conversation.

@i0natan I might be misunderstanding you, but when I have compilation error in my tests cases, the whole test suite fails and none of the tests are even attempted by jest. Is that the pattern you were describing? If so, jest will give a non-empty/non-null failureMessage in the array of test results supplied, but it still lists 0 for each category of tests, and (I think) gets treated like an empty test suite in this library.

I wonder if in reality, any scenario that might have resulted in a JUnit error would just get classified as a failure in jest. If so, you can't really have the same granularity as JUnit does naturally; hopefully I'm wrong about that.

Hopefully I'm helping the conversation!

[palmerj3]

@dangoslen welcome :)

The main issue here is that if a test is run and there is a syntax error, uncaught exception, etc. then Jest currently does not provide enough information to testResultsProcessors (such as jest-junit) to be able to identify it as a failure vs error. So currently, in those situations, they are flagged as failure.

JUnit does recognize error and failure as distinct but, currently, Jest does not provide enough information for me to generically infer this. In some cases you will see an error + stack trace and, it could be argued, that we could infer error in that case. But I am err'ing on the side of caution since Jest does not provide any mechanisms for identifying a stack trace vs simple stderr/console.error output.

Hopefully this helps

[dangoslen]

@palmerj3 thanks for the quick reply!

Thanks for helping to clarify what I was thinking. Bit of a bummer though they don't have that classification, but I guess with scripting some of the errors that normally come from strongly-typed languages aren't as 'important'.

Thanks again for this tool! Its made integrating our pretty small UI resources/tests into a larger maven build really easy with Jenkins to see tests.

[palmerj3]

No worries and thanks for the nice words!

[IanVS]

Seems like the proposal over at jest stalled out? jestjs/jest#5958

I believe this issue was causing my CI to "pass" when really I've had failing erroring tests for who-knows-how-long (we ignore jest's exit status so that other tests always execute as well). The errored tests aren't included in the junit output at all.

[palmerj3]

All tests are included in junit's output. The issue is that jest itself does not differentiate between "error" and "failure". Everything is "failure" for jest unless a test passes or is skipped.

So I think what you're experiencing is something else entirely.

But if you're interested in a WIP PR for jest to do a better job classifying error vs failure then follow this one I submitted today: jestjs/jest#7255

[IanVS]

I think I have the same situation as the OP, who said:

The output .xml file reports 2 tests where both succeeded. Doesn't it make sense to report all the failing tests in the tests suites as failures (Even though they didn't run, they failed to compile)?

This is my situation as well. The tests which errored don't show up in the output at all. Perhaps it's due to a different problem than what you thought it was?

[palmerj3]

Hmm interesting. @IanVS can you put together a repo that reproduces this?

If so I can take a look. I haven't seen this behavior at all personally.