Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to generate failure only report #1666

Closed
chenrui333 opened this issue Jun 20, 2023 · 8 comments · Fixed by #1749
Closed

How to generate failure only report #1666

chenrui333 opened this issue Jun 20, 2023 · 8 comments · Fixed by #1749
Labels
enhancement New feature or request
Milestone
4.1.0

Comments

@chenrui333
Copy link
Contributor

Not sure if this is a feature request or not, I have a use case that I want to send the concise report (i.e. failure test) into slack notification so that folks can determine whether the failures are just flaky tests.

(Right now I am compiling the failure test reports manually)

Let me know if that makes sense.
@chenrui333
Copy link
Contributor Author

cc @jcamiel @fabricereix

@jcamiel
Copy link
Collaborator

jcamiel commented Jun 22, 2023 •
edited
Loading

Sorry for the delay @chenrui333 could you explain a bit further how do you process now? Are you using --report-html or -- json output? How do you build your report? I understand that you want only the failures, is this the case?
We're thinking also about a --report-text option: a plain text report that could be easily greped (filtering failure test for example).

@chenrui333
Copy link
Contributor Author

We're thinking also about a --report-text option: a plain text report that could be easily greped (filtering failure test for example).

yeah, that is what i am looking for, just something simpler that i can compile and send to slack.

@chenrui333
Copy link
Contributor Author

Right now, I am doing lots of manual copy/paste for the hurl failures into the slack threads.

@jcamiel
Copy link
Collaborator

jcamiel commented Jun 27, 2023

Ok, we can start something simple and iterate with your input!

@jcamiel
Copy link
Collaborator

jcamiel commented Jun 27, 2023

What we can also do is to make the HTML report "greppable".

Actually:

<html>
...
<body>
...
<h2>Report</h2>
    <div class="summary">
        <div class="date">Sat, 24 Jun 2023 22:01:01 +0200</div>
        <div class="count">Executed: 5 (100%)</div>
        <div class="count">Succeeded: 2 (40.0%)</div>
        <div class="count">Failed: 3 (60.0%)</div>
    </div>
...

<tr class="failure" data-duration="331" data-status="failure" data-filename="/tmp/test.hurl" data-id="8ac2c1b5-596d-4399-8831-a82302a8233f">
    <td><a href="store/8ac2c1b5-596d-4399-8831-a82302a8233f-timeline.html">/tmp/test.hurl</a></td>
    <td>failure</td>
    <td>0.331</td>
</tr>
<tr class="failure" data-duration="349" data-status="failure" data-filename="/tmp/test.hurl" data-id="a60f6f0e-3d02-4f2f-ab75-07a0dc21726e">
    <td><a href="store/a60f6f0e-3d02-4f2f-ab75-07a0dc21726e-timeline.html">/tmp/test.hurl</a></td>
    <td>failure</td>
    <td>0.349</td>
</tr>
<tr class="success" data-duration="520" data-status="success" data-filename="/tmp/test.hurl" data-id="3e27250b-340f-4969-8a55-a61a8944d994">
    <td><a href="store/3e27250b-340f-4969-8a55-a61a8944d994-timeline.html">/tmp/test.hurl</a></td>
    <td>success</td>
    <td>0.52</td>
</tr>
...
</body>

We could at minima write each tr on a single line:

<html>
...
<body>
...
<h2>Report</h2>
    <div class="summary">
        <div class="date">Sat, 24 Jun 2023 22:01:01 +0200</div>
        <div class="count">Executed: 5 (100%)</div>
        <div class="count">Succeeded: 2 (40.0%)</div>
        <div class="count">Failed: 3 (60.0%)</div>
    </div>
...

<tr class="failure" data-duration="331" data-status="failure" data-filename="/tmp/test.hurl" data-id="8ac2c1b5-596d-4399-8831-a82302a8233f"><td><a href="store/8ac2c1b5-596d-4399-8831-a82302a8233f-timeline.html">/tmp/test.hurl</a></td><td>failure</td><td>0.331</td></tr>
<tr class="failure" data-duration="349" data-status="failure" data-filename="/tmp/test.hurl" data-id="a60f6f0e-3d02-4f2f-ab75-07a0dc21726e"><td><a href="store/a60f6f0e-3d02-4f2f-ab75-07a0dc21726e-timeline.html">/tmp/test.hurl</a></td><td>failure</td><td>0.349</td></tr>
<tr class="success" data-duration="520" data-status="success" data-filename="/tmp/test.hurl" data-id="3e27250b-340f-4969-8a55-a61a8944d994"><td><a href="store/3e27250b-340f-4969-8a55-a61a8944d994-timeline.html">/tmp/test.hurl</a></td><td>success</td><td>0.52</td></tr>

And

$ grep -F '<tr' report/index.html
<tr class="failure" data-duration="331" data-status="failure" data-filename="/tmp/test.hurl" data-id="8ac2c1b5-596d-4399-8831-a82302a8233f"><td><a href="store/8ac2c1b5-596d-4399-8831-a82302a8233f-timeline.html">/tmp/test.hurl</a></td><td>failure</td><td>0.331</td></tr>
<tr class="failure" data-duration="349" data-status="failure" data-filename="/tmp/test.hurl" data-id="a60f6f0e-3d02-4f2f-ab75-07a0dc21726e"><td><a href="store/a60f6f0e-3d02-4f2f-ab75-07a0dc21726e-timeline.html">/tmp/test.hurl</a></td><td>failure</td><td>0.349</td></tr>
<tr class="success" data-duration="520" data-status="success" data-filename="/tmp/test.hurl" data-id="3e27250b-340f-4969-8a55-a61a8944d994"><td><a href="store/3e27250b-340f-4969-8a55-a61a8944d994-timeline.html">/tmp/test.hurl</a></td><td>success</td><td>0.52</td></tr>

We can do something like this for a quick start if it helps you and see --report-txt later.

@fabricereix
Copy link
Collaborator

@chenrui333
we also plan to add the TAP test output (#921) which will be easy to process/grep.
But it does not contain a lot of information.

@chenrui333
Copy link
Contributor Author

that is great!! Subscribed to that issue as well. 😄

@fabricereix fabricereix linked a pull request Jul 11, 2023 that will close this issue
Add TAP report #1749
Merged
@jcamiel jcamiel added this to the 4.1.0 milestone Jul 12, 2023
@jcamiel jcamiel added the enhancement New feature or request label Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
4.1.0
Development

Successfully merging a pull request may close this issue.

Add TAP report Orange-OpenSource/hurl
3 participants
@fabricereix @chenrui333 @jcamiel