Version() was called before having been initialized

[jameshartig]

Describe the problem

Crashed on startup

To Reproduce

Started up CockroachDB after upgrading the binary from v2.1.0-beta.20180910 to v2.1.0-beta.20181008. This happened on 2 of 3 development instances so it doesn't seem to happen all the time.

Expected behavior
Expected not to crash on startup.

Additional data / screenshots

I181015 16:14:15.129244 21 storage/engine/rocksdb.go:574  opening rocksdb instance at "/opt/data/1/cockroachdb"
F181015 16:14:15.653809 85 settings/cluster/settings.go:164  Version() was called before having been initialized
goroutine 85 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc4204a8000, 0xc4204a80c0, 0x3eae100, 0x1c)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:972 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x4788b00, 0xc400000004, 0x3eae152, 0x1c, 0xa4, 0xc420a0cd00, 0x33)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:841 +0x807
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x2fe0b60, 0xc420050078, 0x4, 0x2, 0x0, 0x0, 0xc4206fc070, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:85 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x2fe0b60, 0xc420050078, 0x1, 0x4, 0x0, 0x0, 0xc4206fc070, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x2fe0b60, 0xc420050078, 0xc4206fc070, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:181 +0x6c
github.com/cockroachdb/cockroach/pkg/settings/cluster.(*ExposedClusterVersion).Version(0xc42025d1a8, 0x155dd448a81b1541, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/settings/cluster/settings.go:164 +0x161
github.com/cockroachdb/cockroach/pkg/settings/cluster.(*ExposedClusterVersion).IsActive(0xc42025d1a8, 0x1c, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/settings/cluster/settings.go:199 +0x2b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).augmentMetaLocked(0xc42077d2f8, 0x3e408c5f68bd68b9, 0x370796e0359e8ebe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x155dd4488a4db041, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:371 +0x58
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).augmentMetaLocked(0xc42077d000, 0x3e408c5f68bd68b9, 0x370796e0359e8ebe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x155dd4488a4db041, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:534 +0x11b
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSenderFactory).TransactionalSender(0xc4208b3ee0, 0x1, 0x3e408c5f68bd68b9, 0x370796e0359e8ebe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x155dd4488a4db041, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:479 +0x6da
github.com/cockroachdb/cockroach/pkg/internal/client.NewTxnWithCoordMeta(0x2fe0be0, 0xc420979d10, 0xc4204b4480, 0xc400000000, 0x1, 0x3e408c5f68bd68b9, 0x370796e0359e8ebe, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:145 +0x14c
github.com/cockroachdb/cockroach/pkg/internal/client.NewTxnWithProto(0x2fe0be0, 0xc420979d10, 0xc4204b4480, 0x0, 0x1, 0x3e408c5f68bd68b9, 0x370796e0359e8ebe, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:126 +0x16a
github.com/cockroachdb/cockroach/pkg/internal/client.NewTxn(0x2fe0be0, 0xc420979d10, 0xc4204b4480, 0x0, 0x1, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:116 +0x14e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).prepare(0xc420706000, 0x2fe0be0, 0xc420979d10, 0x2fe4260, 0xc420504a40, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:179 +0x2a4
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).addPreparedStmt(0xc420706000, 0x2fe0be0, 0xc420979d10, 0x0, 0x0, 0x2fe4260, 0xc420504a40, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:122 +0xd4
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPrepare(0xc420706000, 0x2fe0be0, 0xc420979d10, 0x0, 0x0, 0x2fe4260, 0xc420504a40, 0xc420979cb0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:55 +0x144
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc420706000, 0x2fe0be0, 0xc4209799e0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1145 +0x2b63
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx.func1(0xc420706000, 0x2fe0be0, 0xc4209799e0, 0xc42047e540, 0x0, 0xc4202ecd80)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:211 +0x8a
created by github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:200 +0x275

Environment:

• CockroachDB version: v2.1.0-beta.20181008
• Server OS: Linux CentOS 7

[tbg]

Thanks @fastest963! We'll take a look and fix this problem.

[mberhault]

This is a duplicate of #25771. A fix for it is included in the next beta (release due today).
Please try the next beta when it comes out. Until then, you can keep trying to start the nodes until it stops happening. Sorry, it's not a great solution, but the next beta should solve it for good.

Please re-open if you still see this behavior after today's beta.

[tbg]

Is this really a duplicate? It certainly is the same class of bug, but without looking at the code this looks like it might be a separate instance of it.

[mberhault]

How so? This is a startup bug that has the same output and stacktrace (see 2nd stacktrace in the linked issue). While it could technically be a separate bug I still think we should wait for the attempted fix.

[tbg]

Oh, missed that one. Thanks!

[jameshartig]

Woops, I tried searching for the bug but I guess somehow I missed that other one. Looking forward to the next beta!