roachtest: gather cores if requested #91632
Conversation
cucaroach
requested review from
herkolategan and
smg260
and removed request for
a team
|
This change is |
cucaroach
force-pushed
the
gather-cores
branch
2 times, most recently
from
9f30c37
to
3e8d535
Compare
There was a problem hiding this comment.
Reviewed 4 of 4 files at r1, 2 of 2 files at r2, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @cucaroach, @herkolategan, @rytaft, @smg260, and @srosenberg)
pkg/roachprod/vm/gce/utils.go line 171 at r1 (raw file):
echo "kernel.core_pattern=$CORE_PATTERN" >> /etc/sysctl.conf sysctl --system # reload sysctl settings
Seems like this line is removed, was this intentional?
Some roachtest timeout errors are proving difficult to diagnose with just the stacks so add a hook for them to optin to core gathering. Release note: None
cucaroach
force-pushed
the
gather-cores
branch
from
3e8d535
to
add027a
Compare
There was a problem hiding this comment.
Reviewable status:
pkg/roachprod/vm/gce/utils.go line 171 at r1 (raw file):
Previously, yuzefovich (Yahor Yuzefovich) wrote…
Seems like this line is removed, was this intentional?
No nice catch, it isn't strictly necessary for my purposes but who knows what else it was doing so I put it back.
| chmod a+w /mnt/data1/cores | ||
| cat <<'EOF' > /bin/gzip_core.sh | ||
| #!/bin/sh | ||
| exec /bin/gzip -f - > /mnt/data1/cores/core.$1.$2.$3.$4.gz |
There was a problem hiding this comment.
pigz might save a few mins if the core is very large. (May need to apt-get install pigz.)
There was a problem hiding this comment.
For a 300MB core its like 3s vs .3s so wow, that's like a 10x speedup but hardly seems worth it.
There was a problem hiding this comment.
Reviewable status:
pkg/roachprod/vm/gce/utils.go line 171 at r1 (raw file):
Everytime you make changes to your sysctl configuration file, you must reload it to activate the changes.
🤷♂️
|
bors r+ |
There was a problem hiding this comment.
Reviewable status:
pkg/roachprod/vm/gce/utils.go line 171 at r1 (raw file):
Previously, srosenberg (Stan Rosenberg) wrote…
Everytime you make changes to your sysctl configuration file, you must reload it to activate the changes.
🤷♂️
Right but we already editted the /proc/sys/kernel/core_pattern file so we're good, and editting the sysctl.conf makes it stay that way upon reboot. What does the sysctl --system add? At a minimum it ensures the current state and the state after a reboot will be the same, so thats something.
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed (retrying...): |
|
Build failed: |
Release note: None
cucaroach
force-pushed
the
gather-cores
branch
from
add027a
to
b8965c4
Compare
|
bors r+ |
|
Build failed (retrying...): |
|
Build succeeded: |
Relates to #90683 CRDB-20887