Improve helm test output; and provide the means (even if disabled due to flaky tests) for parallel helm test executions.

[jschaul]

1. Allow running helm tests in parallel if desired, using HELM_PARALLELISM=6 (disabled for now until we have fixed some flaky tests which fail more often when tests run in parallel)

2. rework integration test output: logs from test runs will only show if there are any failed tests. Also, the bottom of the output will have a summary of what failed and what didn't; as well as only the failed test lines with a context of +- 10 lines. This should hopefully make it easier to see what went wrong: just scroll to the bottom.

The summary looks like this:

=== tail cargohold: ===

All 21 tests passed (8.45s)
=== tail gundeck: ===

All 33 tests passed (56.60s)
=== tail federator: ===
Finished in 0.6576 seconds
9 examples, 0 failures
=== tail spar: ===
Finished in 397.2779 seconds
553 examples, 0 failures, 65 pending
=== tail brig: ===

2 out of 449 tests failed (123.07s)
=== tail galley: ===

1 out of 414 tests failed (136.33s)
cargohold-integration passed ✅.
gundeck-integration passed ✅.
federator-integration passed ✅.
spar-integration passed ✅.
brig-integration FAILED ❌. pfff...
galley-integration FAILED ❌. pfff...
Tests failed.

Checklist

• Add a new entry in an appropriate subdirectory of changelog.d
• Read and follow the PR guidelines

[smatting]

The relevant log section seems to be missing the line of the failing test.
In this case this line is missing

POST /register - can add team members above fanout limit when whitelisting is enabled:

looks like it got overwritten by

galley-integration: Thread killed by timeout manager

https://concourse.ops.zinfra.io/teams/main/pipelines/wire-server-pr/jobs/integration-tests/builds/1461#L639a4d00:4361

[jschaul]

The relevant log section seems to be missing the line of the failing test. In this case this line is missing

POST /register - can add team members above fanout limit when whitelisting is enabled:

looks like it got overwritten by

galley-integration: Thread killed by timeout manager

https://concourse.ops.zinfra.io/teams/main/pipelines/wire-server-pr/jobs/integration-tests/builds/1461#L639a4d00:4361

Good catch. It happens when there is a line like

some test name:                      {"level": "Info", ....

where the test name is filtered out as it's on the same line as some noisy log lines. I added some sed magic to deal with these cases in the last commit.

[akshaymankar]

I think this hopes that the red colour is reset during the next 10 lines and no other colour is set at that point, which might not be true and mess up rest of the output. I think we should explicitly print a colour reset code after this rg to ensure that doesn't happen.

[jschaul]

this is kind of the end of the ouput for the integration tests. I've not observde any problems with colour so far. But happy to add something. Would a echo -e "\033[0m" line after the rg line do the trick?

[akshaymankar]

Would a echo -e "\033[0m" line after the rg line do the trick?

I think that should work.

[akshaymankar]

Will this directory need to be cleaned up to run the test next time?

[jschaul]

this is just a marker file to avoid gnu parallel complaining about wanting to be cited in academic papers. No need for cleanup if this runs locally.

(the logs-brig and stat-brig files so far remain on disk if you run the integration test locally)

[jschaul]

(update: the script will now clean up these local temporary files by default)

[akshaymankar]

Maybe I should've pointed it out in #3049, but now we don't need the forked helm, maybe we can also get rid of that?

[jschaul]

Maybe I should've pointed it out in #3049, but now we don't need the forked helm, maybe we can also get rid of that?

I already thought of using the fork to get another patch in, this one: helm/helm#11766

But perhaps we can also wait until that is merged (if it ever happens).

[smatting]

No need to attribute to me here ;)

[migueleliasweb]

I already thought of using the fork to get another patch in, this one: helm/helm#11766

But perhaps we can also wait until that is merged (if it ever happens).

I really hope it does eventually get merged! 🙏