roachprod: implement --dry-run and --verbose #99091
Labels
A-roachprod
A-testing
Testing tools and infrastructure
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-testeng
TestEng Team
Comments
srosenberg
added
C-enhancement
|
cc @cockroachdb/test-eng |
srosenberg
added a commit
to srosenberg/cockroach
that referenced
this issue
Dec 21, 2024
Debugging roachprod in a remote (user) environment can be challenging. On several occassions, simply printing all executed (cloud) commands was sufficient to reproduce and root cause an issue. This change implements two mutually exclusive modes, namely `dry-run` and `verbose`. The latter merely logs every command before it's executed. The former suppresses execution of any command which can modify the (cloud) infrastructure; i.e., `dry-run` executes read-only commands; it logs every command, executed or not. Also note that `dry-run` leaves temporary files like VM startup scripts on disk, so that they can be readily examined. Thus, `dry-run` is primarily for dumping _all_ commands without the risk of modifying anything; `verbose` is primarily for live debugging. Release note: None Epic: none Fixes: cockroachdb#99091
srosenberg
added a commit
to srosenberg/cockroach
that referenced
this issue
Dec 21, 2024
Debugging roachprod in a remote (user) environment can be challenging. On several occassions, simply printing all executed (cloud) commands was sufficient to reproduce and root cause an issue. This change implements two mutually exclusive modes, namely `dry-run` and `verbose`. The latter merely logs every command before it's executed. The former suppresses execution of any command which can modify the (cloud) infrastructure; i.e., `dry-run` executes read-only commands; it logs every command, executed or not. Also note that `dry-run` leaves temporary files like VM startup scripts on disk, so that they can be readily examined. Thus, `dry-run` is primarily for dumping _all_ commands without the risk of modifying anything; `verbose` is primarily for live debugging. Release note: None Epic: none Fixes: cockroachdb#99091
srosenberg
added a commit
to srosenberg/cockroach
that referenced
this issue
Dec 21, 2024
Debugging roachprod in a remote (user) environment can be challenging. On several occassions, simply printing all executed (cloud) commands was sufficient to reproduce and root cause an issue. This change implements two mutually exclusive modes, namely `dry-run` and `verbose`. The latter merely logs every command before it's executed. The former suppresses execution of any command which can modify the (cloud) infrastructure; i.e., `dry-run` executes read-only commands; it logs every command, executed or not. Also note that `dry-run` leaves temporary files like VM startup scripts on disk, so that they can be readily examined. Thus, `dry-run` is primarily for dumping _all_ commands without the risk of modifying anything; `verbose` is primarily for live debugging. Release note: None Epic: none Fixes: cockroachdb#99091
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, roachprod debugging consists of either trying to match the output to the code, or executing (cloud) commands to observe their effects. There is also some bootstrapping "magic" (e.g., ssh setup) which happens without much of a trace in the logged output. Consequently, even an experienced roachprod user may end up spending considerable time to diagnose an infra. issue.
Both the
--dry-run, and the--verbosemode should make it easier to troubleshoot roachprod issues. By logging the commands, the "magic" usually becomes self-explanatory. Furthermore,--dry-runwould considerably speed up debugging without having to apply actual infra. changes.Jira issue: CRDB-25697
The text was updated successfully, but these errors were encountered: