colexec: v24.1.2: temp storage error is incorrectly propagated as InternalError

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6622546216/?referrer=webhooks_plugin

Panic Message:

error.go:61: unexpected error from the vectorized engine: ×
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:61
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:267
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*partitionerToOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/utils.go:70
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.getNext_false
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go:540
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*hashAggregator).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go:90
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*hashBasedPartitioner).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/hash_based_partitioner.go:592
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:223
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).processNextBatch
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:667
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:591
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:629
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:135
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:586
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:831
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.(*vectorizedFlowCreator).accumulateAsyncComponent.func2.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:734
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1222
Wraps: (4) unexpected error from the vectorized engine
Wraps: (5) ×
  | -- cause hidden behind barrier
  | ×
  | (1) ×
  | Error types: (1) *errors.errorString
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.withPrefix (5) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: ×
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:61: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s#L1221-L1223

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 733 to 735 in 7e81be6

	defer wg.Done()
	run(ctx, flowCtxCancel)
	}()

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 830 to 832 in 7e81be6

	runRouter := func(ctx context.Context, _ context.CancelFunc) {
	router.Run(logtags.AddTag(ctx, "hashRouterID", streamIDs))
	}

cockroach/pkg/sql/colflow/routers.go

Lines 585 to 587 in 7e81be6

	// well for more fine-grained control of error propagation.
	if err := colexecerror.CatchVectorizedRuntimeError(func() {
	r.Input.Init(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 134 to 136 in 7e81be6

	}()
	operation()
	return retErr

cockroach/pkg/sql/colflow/routers.go

Lines 628 to 630 in 7e81be6

	if err := colexecerror.CatchVectorizedRuntimeError(processNextBatch); err != nil {
	r.cancelOutputs(ctx, err)

cockroach/pkg/sql/colexecerror/error.go

Lines 134 to 136 in 7e81be6

	}()
	operation()
	return retErr

cockroach/pkg/sql/colflow/routers.go

Lines 590 to 592 in 7e81be6

	processNextBatch := func() {
	done = r.processNextBatch(ctx)
	}

cockroach/pkg/sql/colflow/routers.go

Lines 666 to 668 in 7e81be6

	func (r *HashRouter) processNextBatch(ctx context.Context) bool {
	b := r.Input.Next()
	n := b.Length()

cockroach/pkg/sql/colflow/stats.go

Lines 125 to 127 in 7e81be6

	// execution time (e.g. in the statement bundle).
	err := colexecerror.CatchVectorizedRuntimeError(bic.next)
	bic.stopwatch.Stop()

cockroach/pkg/sql/colexecerror/error.go

Lines 134 to 136 in 7e81be6

	}()
	operation()
	return retErr

cockroach/pkg/sql/colflow/stats.go

Lines 117 to 119 in 7e81be6

	func (bic *batchInfoCollector) next() {
	bic.batch = bic.Input.Next()
	}

cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go

Lines 222 to 224 in 7e81be6

	d.diskBackedOpInitialized = true
	return d.diskBackedOp.Next()
	}

cockroach/pkg/sql/colexec/colexecdisk/hash_based_partitioner.go

Lines 591 to 593 in 7e81be6

	case hbpProcessingUsingMain:
	b := op.inMemMainOp.Next()
	if b.Length() == 0 {

https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go#L89-L91
https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go#L539-L541

cockroach/pkg/sql/colexec/colexecdisk/utils.go

Lines 69 to 71 in 7e81be6

	if err != nil {
	colexecerror.InternalError(err)
	}

cockroach/pkg/sql/colexecerror/error.go

Lines 266 to 268 in 7e81be6

	func InternalError(err error) {
	panic(newInternalError(err))
	}

GOROOT/src/runtime/panic.go#L769-L771

cockroach/pkg/sql/colexecerror/error.go

Lines 60 to 62 in 7e81be6

	if code := pgerror.GetPGCode(retErr); code == pgcode.Uncategorized {
	retErr = errors.NewAssertionErrorWithWrappedErrf(
	retErr, "unexpected error from the vectorized engine",

src/runtime/asm_arm64.s in runtime.goexit at line 1222
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.(*vectorizedFlowCreator).accumulateAsyncComponent.func2.1 at line 734
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1 at line 831
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run at line 586
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 135
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1 at line 629
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 135
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1.1 at line 591
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).processNextBatch at line 667
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 126
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 135
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 118
pkg/sql/colexec/colexecdisk/disk_spiller.go in pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next at line 223
pkg/sql/colexec/colexecdisk/hash_based_partitioner.go in pkg/sql/colexec/colexecdisk.(*hashBasedPartitioner).Next at line 592
bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go in pkg/sql/colexec.(*hashAggregator).Next at line 90
bazel-out/k8-opt/bin/pkg/sql/colexec/hash_aggregator.eg.go in pkg/sql/colexec.getNext_false at line 540
pkg/sql/colexec/colexecdisk/utils.go in pkg/sql/colexec/colexecdisk.(*partitionerToOperator).Next at line 70
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 267
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 61

Tags

Tag	Value
Command	server
Environment	v24.1.2
Go Version	go1.22.3 X:nocoverageredesign
Platform	linux arm64
Distribution	CCL
Cockroach Release	v24.1.2
Cockroach SHA	7e81be6
# of CPUs	12
# of Goroutines	589

Jira issue: CRDB-50892

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/sql-queries

Sentry issue cause: pkg/sql/colexecerror/error.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}