Add JUnit XML Report Output

[atcol]

This is for discussion of #325.

I added --junit ${FILENAME} for outputting test results in the JUnit XML format. I used the json flag name as I felt it was similar to html.

Are there any issues with doing it this way @fabricereix ?

[jcamiel]

Hi,

The Junit report option should work like html and json report: if the files already exists, it will be updated with the new test results.

Beside this pr, we need to think about how --html, --json, --junit option works: does the option require a file output, or use stdout ?

Currently:

• --html: requires a directory, produces a bunch of files, append to existing directory if exist.
• --json: requires a file, produces a file, append to existing file if exist. After working on Use --json parameter to write JSON to stdout #283, we think it should be a good idea to use sdtout to produce the json report. Does we keep also the possibility to use a file?: --json without a file name (or --json -) uses sdtout, whereas --json myfile uses this file.
• --junit (this PR): should work like --json : requires a file, produces a file, append to existing file. But like --json, stdout should be used ?

It could be usefull also to write in the man/doc if option use stdout or sdrerr (for instance, it's not clear if --progress or --summary uses stdout or stderr)

[atcol]

Not sure I follow. Each individual run of hurl with --junit X is separate - i.e. run 1's results might differ entirely to run 2. Why would we append to the report file in such cases?

[jcamiel]

I think it would be better to be consistent with the existing options:

• --json, --html produces a kind of report. If there is a previous report, the newly created is appended with the existing one,

• --summary produces a simple plain report considering all Hurl files that have been run (see https://hurl.dev/docs/running-tests.html):

  $ hurl --summary hello.hurl assert_json.hurl
  Hello World![
  { "id": 1, "name": "Bob"},
  { "id": 2, "name": "Bill"}
  ]
  --------------------------------------------------------------------------------
  Executed:  2
  Succeeded: 2 (100.0%)
  Failed:    0 (0.0%)
  Duration:  134ms
  

In JUnit format, there are:

• testsuites: the aggregated result of all junit testfiles
• testsuite: the output from a single TestSuite
• testcase: the results from executing a test method

A possible implementation for Hurl could be:

• testsuites: the aggregated result of all Hurl test files
• testsuite: the output from a single Hurl file
• testcase: the results from executing a single Hurl Entry

[atcol]

A possible implementation for Hurl could be:

testsuites: the aggregated result of all Hurl test files
testsuite: the output from a single Hurl file
testcase: the results from executing a single Hurl Entry

This is what my implementation does.

[jcamiel]

Ok, that's very good.

But, if I run:

hurl --output /dev/null --json report.json test-a.hurl test-b.hurl

The report.json reports will contains test-a + test-b.

If I run again with another file:

hurl --output /dev/null --json report.json test-c.hurl

The report.json reports will contains test-a + test-b + test-c.

--htmlis also working like this: tests are appended to existing file.

With the current PR:

hurl --output /dev/null --junit report.xml test-a.hurl test-b.hurl

The report.xml reports will contains test-a + test-b (as testsuite, with is good)

But if I run again

hurl --output /dev/null --junit report.xml test-c.hurl

The report.xml reports will contains only test-c

So the PR junit option is not consistent with existing option (--jsonand --html). These options are all related to a kind of report and should be consistent.

[atcol]

I see what you mean. As I understand it the average use case for JUnit reports would have these results replaced as I've currently implemented it, but I certainly agree with the consistency argument and I don't see any harm in implementing it as you & other(s) have suggested. I'll update the branch tonight.

[fabricereix]

I think more of a Hurl file as the basic Testcase (rather than the Entry).

An entry is used to group a request/response and is not directly related to a test.
For example, a request may be necessary to fetch a token (Login, CSRF,..) in order to test endpoints, but is not part of the tests as such.

[fabricereix]

We might need to be more explicit between an output written to stdout and an output that is written an appendable "report" file.

Currently we have:

• stdout output: Response body (curl-like)
• report file: HTML, JSON

The summary output is currently written to stderr, but it could be one of of the stdout output (#328).
The JSON could be also a type of output (#283)

As we discussed, the Junit output is more like the report file.
We could maybe distinguish the report files with options like --html-report and junit-report

[atcol]

We could maybe distinguish the report files with options like --html-report and junit-report

Sounds like --report json (with support for html or junit) would be more appropriate/cleaner.

[fabricereix]

We also need to give the filename.

hurl --html-report /tmp/html --junit-report /tmp/report.xml tests/*.hurl

[fabricereix]

We reviewed the options in #330.
--report-junit FILE can be used to generate the JUnit report.