ccl/kvccl/kvfollowerreadsccl: TestBoundedStalenessDataDriven failed

[cockroach-teamcity]

ccl/kvccl/kvfollowerreadsccl.TestBoundedStalenessDataDriven failed with artifacts on master @ ff839799faaf1e9ebda21e09e959e47f697d821a:

=== RUN   TestBoundedStalenessDataDriven
    test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/8c04cb0c40c192808857b5e756290a7c/logTestBoundedStalenessDataDriven3294506660
    test_log_scope.go:82: use -show-logs to present logs inline
    boundedstaleness_test.go:434: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/8c04cb0c40c192808857b5e756290a7c/logTestBoundedStalenessDataDriven3294506660
--- FAIL: TestBoundedStalenessDataDriven (7.69s)
=== RUN   TestBoundedStalenessDataDriven/single_row
    test_server_shim.go:154: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14970/execroot/com_github_cockroachdb_cockroach/bazel-out/s390x-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:5:
          exec-system-tenant
          SET CLUSTER SETTING kv.closed_timestamp.target_duration = '1s';
          ----
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14970/execroot/com_github_cockroachdb_cockroach/bazel-out/s390x-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:9:
          exec
          CREATE TABLE t(pk INT PRIMARY KEY) WITH (schema_locked = false);
          INSERT INTO t VALUES (1);
          ----
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14970/execroot/com_github_cockroachdb_cockroach/bazel-out/s390x-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:18:
          query idx=2 wait-until-match
          SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('1μs') WHERE pk = 1
          ----
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/14970/execroot/com_github_cockroachdb_cockroach/bazel-out/s390x-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:25:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '1μs') WHERE pk = 1
        
        expected:
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
        
        found:
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read
--- FAIL: TestBoundedStalenessDataDriven/single_row (7.67s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-55086

[cockroach-teamcity]

ccl/kvccl/kvfollowerreadsccl.TestBoundedStalenessDataDriven failed with artifacts on master @ e1d612210b299df836bdbdb95b5e6c979529a813:

    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:141:
          exec-system-tenant
          SET CLUSTER SETTING kv.closed_timestamp.target_duration = '1hr';
          ----
    datadriven.go:352: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:145: still running after 10.004855s
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:145:
          exec
          ALTER TABLE t ADD COLUMN new_col INT NOT NULL DEFAULT 2
          ----
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:151:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('10s') WHERE pk = 1
          ----
          1 2
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:158:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '10s') WHERE pk = 1
          ----
          1 2
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:165:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('10s', false) WHERE pk = 1
          ----
          1 2
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:172:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '10s', false) WHERE pk = 1
          ----
          1 2
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    datadriven.go:352: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:184: still running after 10.003845128s
    datadriven.go:352: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:184: still running after 20.004143791s
    datadriven.go:352: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:184: still running after 30.005023647s
    datadriven.go:352: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17294/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:184: still running after 40.008759084s
    boundedstaleness_test.go:394: condition failed to evaluate within 45s: from boundedstaleness_test.go:419: not follower reads found:
        events (0 found):
--- FAIL: TestBoundedStalenessDataDriven/single_row (64.40s)

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

ccl/kvccl/kvfollowerreadsccl.TestBoundedStalenessDataDriven failed with artifacts on master @ 14ff833383127fa428b913c5281a8a6c862e2a41:

=== RUN   TestBoundedStalenessDataDriven
    test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/18f40673c89ae073fd9b4688a4790099/logTestBoundedStalenessDataDriven1739197817
    test_log_scope.go:82: use -show-logs to present logs inline
    boundedstaleness_test.go:434: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/18f40673c89ae073fd9b4688a4790099/logTestBoundedStalenessDataDriven1739197817
--- FAIL: TestBoundedStalenessDataDriven (12.20s)
=== RUN   TestBoundedStalenessDataDriven/single_row
    test_server_shim.go:154: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17082/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:5:
          exec-system-tenant
          SET CLUSTER SETTING kv.closed_timestamp.target_duration = '1s';
          ----
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17082/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:9:
          exec
          CREATE TABLE t(pk INT PRIMARY KEY) WITH (schema_locked = false);
          INSERT INTO t VALUES (1);
          ----
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17082/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:18:
          query idx=2 wait-until-match
          SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('1μs') WHERE pk = 1
          ----
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:315: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/17082/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:25:
          query idx=2
          SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '1μs') WHERE pk = 1
        
        expected:
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
        
        found:
          1
          events (1 found):
           * event 1: colbatchscan trace on node_idx 2: local read
--- FAIL: TestBoundedStalenessDataDriven/single_row (12.16s)

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!