detailed test step information in the test result

[EdStrickland]

HI,
our team is currently running BDD and wants to generate a cucumber report. I've written an Karma plugin but found that test steps information weren't in the result.
Is there anyway I could get such information? If not, is there a plan of expossing it?

[matz3]

What do you mean with test steps?

[EdStrickland]

@matz3 I mean the results of all the check points

[matz3]

I'm still not sure what you mean.

This plugin offers two modes.

One is "script" which should not influence the way test results are recorded. This needs to be done in combination with a karma plugin like karma-qunit.

The other one is "html", which has built-in reporting for the QUnit framework, which also means no other test frameworks can be used in this mode. Here it could be that our code doesn't report results back to karma in a proper way, although I'm not aware of any known issues.

Are you able to provide a reproducible example project?

[EdStrickland]

@matz3 I cannot provide our project, so I will need to write a demo. I can only explain the issue as mush as I can while I wrote it.

So when you write a karma-reporter plugin, you can rewrite the callback function onSpecComplete, which receives an object 'result' that gives you information of the test case that has been completed. However, the information doesn't contain which check point the test passed and which isn't.

[matz3]

Got it, thanks 👍
I can have a deeper look into it once a demo/example is provided.

[EdStrickland]

@matz3 I just looked in to the karma-ui5 source code. It seems in 'client/browser.js' the Qunit.log() won't send the result to karma. Is it possible that you provide the result out of the input 'details' and send it to karma?

[matz3]

Currently it is collecting all errors and then posting them within QUnit.testDone to karma.
That's also what the karma-qunit plugin does.

Feel free to submit a PR.

[EdStrickland]

@matz3 Hi, I submitted PR#161 but the pipeline seems not running, is it expected to happen?

[EdStrickland]

#161

[EdStrickland]

Hi,
Is anyone available to help to review my PR? It's been nine days since last updated.

[matz3]

Sorry for the delay. I've added comments to your PR.
But before merging I would still like to fully understand this use case, also in order to properly document it.

[EdStrickland]

Thanks for the review. I'm changing the code accordingly.
About the use case, would you like to know our scenario, or the scenario for the code in general?

[matz3]

I would like to understand how the adopted report is used in combination with cucumber and why those assertions should also be reported as a "test".

[EdStrickland]

Hi @matz3 ,
Sorry for the delay, I was busy working on another feature of our product.

We've just introduced Behavior Driven Development into our development process. During the development, integration tests, OPA tests, would be generated by 'feature files', which are files that describe the expected behaviors of the product written in Gherkin. In the files, the behaviors would be broken down into features, features broken down into several scenarios, and scenarios into steps.

For instance, a 'feature' of the app is Project Creation. A user can create different types of projects; hence there are various 'scenarios'. The actions that a user interacts with the app are 'steps'. Therefore In the integration tests, the assertions suggest the 'steps'.
When the integration tests are run, a cucumber report is expected to be generated, reporting how the tests are running. The information on 'steps' results are as important as 'scenarios' because it shows how the scenario is designed to function and makes showing the result makes it easier to do troubleshooting.
Therefore, we would like to report the result of the assertions when running OPA tests.

[matz3]

Ok thanks, I got it now. Sounds reasonable to have this, although it should ideally also be added to the karma-qunit plugin.

[EdStrickland]

Hi @matz3 , integration tests has been finished. Would you please take a look?

[jiangyichun]

Hi @matz3,

We also need this enhancement very much, do you think you can help adapt and merge the PR: #161? Thanks very much.

Best regards,
Eric Jiang