roachtest: disk-stalled/fuse/log=false,data=true failed

[cockroach-teamcity]

roachtest.disk-stalled/fuse/log=false,data=true failed with artifacts on release-22.2 @ 4014ae25a2946464984b00d89411923303a1f634:

test artifacts and logs in: /artifacts/disk-stalled/fuse/log=false_data=true/run_1
(test_runner.go:938).runTest: test timed out (10h0m0s)
(cluster.go:1977).Run: output in run_100639.318181288_n1_charybdefsnemesis-cl: charybdefs-nemesis --clear returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_100639.324312367_n1_charybdefsnemesis-cl.log: exit status 131
(cluster.go:1977).Run: cluster.RunE: context canceled
(cluster.go:1977).Run: cluster.RunE: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-25568

[jbowens]

10:06:38 disk_stall.go:220: [w5] 489929.67 total transactions committed after stall
10:06:38 disk_stall.go:221: [w5] pre-stall tps: 1212.56, post-stall tps: 1166.50
10:06:38 disk_stall.go:228: test status: counting kv rows
10:06:39 disk_stall.go:231: [w5] Scan found 462504 rows.
19:47:54 test_impl.go:339: test failure #1: full stack retained in failure_1.log: (test_runner.go:938).runTest: test timed out (10h0m0s)

What was the test doing for 10h? Also, I thought this test had a much shorter timeout.

It was stuck removing the delay:

run_100639.318181288_n1_charybdefsnemesis-cl: 10:06:39 cluster.go:290: > charybdefs-nemesis --clear
./recipes: line 7: 37185 Quit                    (core dumped) python2 recipes.py "$@"
run_100639.318181288_n1_charybdefsnemesis-cl: 19:47:55 cluster.go:1999: > result: charybdefs-nemesis --clear returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_100639.324312367_n1_charybdefsnemesis-cl.log: exit status 131
(1) charybdefs-nemesis --clear returned
  | stderr:
  | ./recipes: line 7: 37185 Quit                    (core dumped) python2 recipes.py "$@"
  |
  | stdout:
Wraps: (2) Node 1. Command with error:
  | ```
  | charybdefs-nemesis --clear
  | ```
Wraps: (3) COMMAND_PROBLEM
Wraps: (4) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).errWithDebug
  |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:128
  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*remoteSession).Run.func1
  |     github.com/cockroachdb/cockroach/pkg/roachprod/install/session.go:158
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (5) ssh verbose log retained in ssh_100639.324312367_n1_charybdefsnemesis-cl.log
Wraps: (6) exit status 131
Error types: (1) *cluster.WithCommandDetails (2) *hintdetail.withDetail (3) errors.Cmd (4) *withstack.withStack (5) *errutil.withPrefix (6) *exec.ExitError

Verbose log:

debug1: Sending command: export ROACHPROD=1 GOTRACEBACK=crash && bash -c "charybdefs-nemesis --clear"
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: read<=0 rfd 5 len 0
debug2: channel 0: read failed
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 5 efd 7 [write])
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82
debug3: send packet: type 80
debug3: receive packet: type 82

with this back and forth continuing until the test timed out.

Not sure what happened during the attempt to unstall the FUSE filesystem. If the FUSE variants prove too flaky, we could consider removing them and relying on the dmsetup and cgroup variants. I'm going to close this out as a flake in the test.

[nicktrav]

Thanks for digging in! I also added X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue (we track these types of failures now).