release-22.1: roachprod: capture ssh logs, roachtest: report timeout failures correctly#96979
Merged
smg260 merged 3 commits intocockroachdb:release-22.1from Mar 8, 2023
Merged
Conversation
|
Thanks for opening a backport. Please check the backport criteria before merging:
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
Add a brief release justification to the body of your PR to justify this backport. Some other things to consider:
|
Member
Currently we do not capture SSH logs in the event of a command failure, which can be useful in debugging issues, transient or otherwise. This commit enables logging via the ssh switch -vvv and specifying a log filename, to be stored under an ssh/ directory in the test log root. The debug file is deleted upon successful (zero) exit of the command, but preserved for non-zero exits for further inspection. Additionally, - The name of the log is consistent with the corresponding run log and encodes a node number and timestamp. - SSH sessions must now be initialised with the command itself to re-inforce its single use nature. - Debug friendly command names can optionally be specified to influence the name of the run/ssh logs. - Retry options can optionally be omitted from any call to ParallelE to disable retries Release note: None Epic: CRDB-21386
00e5a8f to
be23fb0
Compare
added 2 commits
March 7, 2023 18:40
Wait commands are issued every 500ms returning a non zero exit code until nodes have started. This results in a large number of ssh debug logs during cluster creation. Also adopts functional options. Release note: None Epic: none
Timeout failures are recorded at actual timeout, with subsequent failures secondary. `addFailure` accepts a depth parameter and no longer includes context cancellation, which is done separately. Epic: none Fixes: cockroachdb#91237 Release note: None
be23fb0 to
76c356a
Compare
renatolabs
approved these changes
Mar 7, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport:
Please see individual PRs for details.
/cc @cockroachdb/release
Epic: None
Release note: None
Release justification: test only change