Skip to content

release-22.1: roachprod: capture ssh logs, roachtest: report timeout failures correctly#96979

Merged
smg260 merged 3 commits intocockroachdb:release-22.1from
smg260:backport22.1-92667-95762-96743
Mar 8, 2023
Merged

release-22.1: roachprod: capture ssh logs, roachtest: report timeout failures correctly#96979
smg260 merged 3 commits intocockroachdb:release-22.1from
smg260:backport22.1-92667-95762-96743

Conversation

@smg260
Copy link
Copy Markdown
Contributor

@smg260 smg260 commented Feb 10, 2023

Backport:

Please see individual PRs for details.

/cc @cockroachdb/release

Epic: None
Release note: None
Release justification: test only change

@blathers-crl
Copy link
Copy Markdown

blathers-crl Bot commented Feb 10, 2023

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues or test-only changes.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Currently we do not capture SSH logs in the event of a
command failure, which can be useful in debugging issues,
transient or otherwise.

This commit enables logging via the ssh switch -vvv and
specifying a log filename, to be stored under an ssh/
directory in the test log root. The debug file is deleted
upon successful (zero) exit of the command, but preserved
for non-zero exits for further inspection.

Additionally,
- The name of the log is consistent with the
corresponding run log and encodes a node number and
timestamp.
- SSH sessions must now be initialised with the
command itself to re-inforce its single use nature.
- Debug friendly command names can optionally be
specified to influence the name of the run/ssh logs.
- Retry options can optionally be omitted from any call
to ParallelE to disable retries

Release note: None
Epic: CRDB-21386
@smg260 smg260 force-pushed the backport22.1-92667-95762-96743 branch from 00e5a8f to be23fb0 Compare March 7, 2023 18:11
Miral Gadani added 2 commits March 7, 2023 18:40
Wait commands are issued every 500ms returning a non zero exit
code until nodes have started. This results in a large number
of ssh debug logs during cluster creation.

Also adopts functional options.

Release note: None
Epic: none
Timeout failures are recorded at actual timeout, with
subsequent failures secondary.

`addFailure` accepts a depth parameter and no longer
includes context cancellation, which is done separately.

Epic: none
Fixes: cockroachdb#91237
Release note: None
@smg260 smg260 force-pushed the backport22.1-92667-95762-96743 branch from be23fb0 to 76c356a Compare March 7, 2023 18:40
@smg260 smg260 marked this pull request as ready for review March 7, 2023 20:34
@smg260 smg260 requested review from a team as code owners March 7, 2023 20:34
@smg260 smg260 requested review from renatolabs and srosenberg and removed request for a team March 7, 2023 20:34
@smg260 smg260 requested review from a team and removed request for a team March 8, 2023 00:39
@smg260 smg260 merged commit 3504ebb into cockroachdb:release-22.1 Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants