Step reports export

[kreuzberger]

Cucumber supports the output of test results as an machine readable json file.
This could be used to generate test reports.
There is a format description here https://github.com/cucumber/messages

There could be several approaches:

• go a own way and just output an own, custom json format
• create a cucumber/messages compatible plugin format to add the library
• what ever i didn't think of.

Due to the different needs of such a test report and to keep the project dependency free the first solution could be the most acceptable, could be "similar" to the json from cucumber messages.
Currently only the feature and step / outlines /examples execution and output are of interested for me (to generate an QS Deparment friendly test report) . Timings and other stuff not.

[kreuzberger]

This is just an "enhancement" with lowest priority. Just as an idea to support custom formatters

[ThoSe1990]

Yes, I have this on my mind. This actually was the motivation to rewrite the interpreter and introduce the visitor pattern, in order to set the foundation for reporting mechanisms. But I'll need some time to do this and I'm rather busy at the moment. I'll leave this issue open.

[ThoSe1990]

I'm about to work on this, can not promise when its finished (I'm moving to a new apartment this week).

However, I played araound with cucumber-js and this would be my first draft, feel free to modify if you need more fields (if you need execution time for each step please lemme know, it isn't a big deal to implement):

[
  {
    "description": "",
    "elements": [
      {
        "description": "",
        "id": "feature-name;scenario-name",
        "keyword": "Scenario",
        "line": 4,
        "name": "...",
        "steps": [
          {
            "arguments": [
              // Datatables or Docstrings
            ],
            "keyword": "Given ",
            "line": 5,
            "name": "...",
            "match": {
              "location": "features/step_definitions/file:line"
            },
            "result": {
              "status": "passed"
              // if execution time is needed we can add it 
            }
          }
          // next step
        ],
        "tags": [],
        "type": "scenario"
      }
      // next scenario
    ],
    "id": "feature-name",
    "line": 2,
    "keyword": "Feature",
    "name": "...",
    "tags": [],
    "uri": "features/example.feature"
  }
  // next feature
]

[kreuzberger]

looks good for a first draft. I do not need execution time...

[ThoSe1990]

Quick update since this is 4+ weeks old: I'm on it, this weekend or beginning of next week I'll finish this. I have all the data for the report. Writing it to json is still missing but most work is done.

[ThoSe1990]

merged #60

If you need changes or additional fields in the json, just open another issue

[kreuzberger]

According to the issues original description, the generated json file could not be read by "standard" cucumber json reports e.g. like for jenkins-ci, cause it is not compatible with https://github.com/cucumber/messages.
Should this reached by post processing the file with an conversion tool?

[ThoSe1990]

Let me do some research, I think an extra conversion tool is a bit unnecessary. Then we can modify the current json export in order to match. I honestly didn't go through the cucumber message repo from the original issue after I posted this json proposal later.

I never did post processing and adopted to cucumber-js and python behave and since they are a bit different I was not aware that there is a official layout

[kreuzberger]

I agree, some research should be made to clarify WHAT should be achived / could be achieved.

Whats my main intention? Cause Cucumber / BDD is for me mainly used in customer acceptance test, the test reports should be visualized / printed in a customer readable / acceptable / configurable format. This is what the json file is good for.
I currently do not know HOW to present the reports to the customer. Just as an Report / HTML from the jenkins CI or as a separate document (pdf/html) like ones generated e.g. like from from https://github.com/rmpestano/cukedoctor (in my case more as a sphinx-plugin).

The output from behave seems to be different, see (behave/behave#267), i assumed they are compatible. Due to the ticket history, it seems that the json schema was introduced not long ago.

https://github.com/cucumber/messages seems to be the streaming version of json, thereas https://github.com/cucumber/json-formatter writes those messages down into a file. So for our intention only parts of the message schema must be implemented (e.g. no states like tests starting, running, finished etc).

[kreuzberger]

Due to the documentations the json from cucumber.js should be in the https://github.com/cucumber/messages format in the versions above 7.0.0

[kreuzberger]

I was wrong assuming the json is not in valid format.
The only thing thats wrong is the mapped "result" of the steps. If changed from "results" to "result" the scenarios could get mapped.

Testet with cucumber-html-reporter

[ThoSe1990]

Thanks a lot for all this input. I changed the results field to result (will merge it after pipelines pass).

Perhaps you can give me another feedback if you can consume the json file now. If so I'd close the PR here.

I double check with the message repo to find any inconsistencies with my implementation and try to make it as complete as possible.

[ThoSe1990]

I merged the change from "results" to "result" #61

[kreuzberger]

The version from the main branch now works with the cucumber-html-reporter.
For me this is currently enough. 👍

[ThoSe1990]

Great 🎉 was rather unfortunate that I accidentally put the plural into the json. Anyway. thanks for pointing that mistake out. If you need additional fields in the json, just open another issue.

[kreuzberger]

pytest-bdd and pytest-bdd-ng also use this plural form, maybe this has changed in the definition of the json schema.
But i think the "singular" form is right due to jsonschema/TestRunHookFinished.json from cucumber/messages.

[kreuzberger]

An additional question: why is the output exclusive?
I am running the test as part of ctest, and when it fails the output from ctest with "normal" reporting on commandline is quite good to identfy. The reports in json file should be helpful for ci integration/reporting and maybe generation of a test protocol itself. Would it be much effort to enable the normal output (not -quiet) AND the generation of the json file itself?

[ThoSe1990]

Its no effort, i just can remove the flag. I don't really have an opinion on that. My thinking was that when the json report is needed, most likely this will be postprocessed (or piped via stdout) which means the regular output is not needed.

But I'll remove the implicit --quiet and if no further output is needed, just pass -q on the command line.

[ThoSe1990]

Merged ✅ #62