roachtest: decommission/nodes=4/duration=1h0m0s failed

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1028,test_runner.go:927: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-16831

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ 377b113bd7e7b679c55a0f93e0d8ab5dbb6f2e97:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1028,test_runner.go:927: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[aayushshah15]

Going to assign this to myself in case it's connected to #80993

[aayushshah15]

Some discussion here: https://cockroachlabs.slack.com/archives/G01G8LK77DK/p1655832081495559.

[aayushshah15]

It looks like #82988 uncovered a bug by removing the implicit timeout after which cockroach quit force-killed the process. Based on discussion with @knz, this implies a bug with node shutdown where, for instance, some stopper task might not be set up to watch for the stopper quiescing.

The next step is to grab goroutine dumps from this decommissioned node to see what is stuck - TODO for @aayushshah15

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ fc7ae339c3f85ad2a2b28643e47d7c20768fe237:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ b47dc374417fe4ba83570784040da5650238b30e:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ a611410e3babe97a14152381b89060e1bafa2e27:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.decommission/nodes=4/duration=1h0m0s failed with artifacts on master @ 7f8fef4c3f24ec2b6f9a3db1eb38756da68b3d5a:

test artifacts and logs in: /artifacts/decommission/nodes=4/duration=1h0m0s/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!