tests/e2e: more debug info on dumps #357
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kevsecurity
reviewed
Aug 23, 2022
willfindlay
force-pushed
the
pr/willfindlay/even-better-dumps
branch
2 times, most recently
from
6240623 to
b04dd26
Compare
kevsecurity
approved these changes
Aug 23, 2022
kkourt
approved these changes
Aug 23, 2022
There were a few leaked goroutines in e2e framework due to the context never getting canceled at the end of the tests, breaking some of our assumptions. Refactor the runner to wrap the testenv and cancel the context at the end of the test. Signed-off-by: William Findlay <[email protected]>
In cases where the agent crashes during a test, we would be unable to retrieve metrics at the end (since the metrics server is now offline). This means that we lose out on some valuable debugging information to deduce what went wrong. To rectify this shortcoming, introduce some new logic to regularly dump Tetragon metrics during the test at a specific interval. With these changes in place, we now at least have a recent snapshot of metrics regardless of whether the metrics server can be contacted at the end of a test. Signed-off-by: William Findlay <[email protected]>
Create a new RunCommand helper and factor it out of the main body of the bpftool dump subroutine. This will enable us to reuse the same basic logic to run commands elsewhere with a nice dump of stdout and stderr. Signed-off-by: William Findlay <[email protected]>
In case the Tetragon pod crashes during a test, it's useful to grab the output of kubectl describe to help understand why. So let's add this to the test dump on failure. Signed-off-by: William Findlay <[email protected]>
It's useful to get a summary of pods in our cluster when a test fails so we can figure out if the reason for the failure might be incidental -- for example in cases where our workload fails to run correctly. Signed-off-by: William Findlay <[email protected]>
willfindlay
force-pushed
the
pr/willfindlay/even-better-dumps
branch
from
b04dd26 to
6db73d0
Compare
|
@kkourt I addressed your comments. Also realized at the same time that we were never actually cancelling our context in the tests, so I had to refactor the runner a bit (it's in the new first commit). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This series adds a bunch more debug info to e2e test dumps. See commits.