tidb bootstrap takes a long time

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. deploy a cluster with 3 tidb nodes
2. import data about 25w partitions and 5w tables
3. restart tidb

2. What did you expect to see? (Required)

tidb can restart immediately

3. What did you see instead (Required)

it's takes a long time to bootstrap

tidb-0

tidb-1

tidb-2

4. What is your TiDB version? (Required)

[10:55:48]TiDB root:INFORMATION_SCHEMA> select tidb_version();
+-----------------------------------------------------------+
| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v7.5.0-test                              |
| Edition: Community                                        |
| Git Commit Hash: fa75dbd363128183665c03e06af67ed9066c1d98 |
| Git Branch: heads/refs/tags/v7.5.0-test                   |
| UTC Build Time: 2023-11-02 06:43:24                       |
| GoVersion: go1.21.3                                       |
| Race Enabled: false                                       |
| Check Table Before Drop: false                            |
| Store: tikv

[aytrack]

related #48239

[tangenta]

It is blocked by show cluster config from

tidb/pkg/ddl/ingest/backend_mgr.go

Line 65 in 89ec3d3

isRaftKV2, err := util.IsRaftKv2(ctx, sctx)

0  0x0000000001c2f86e in runtime.gopark
	at /usr/local/go/src/runtime/proc.go:399
1  0x0000000001c414b8 in runtime.goparkunlock
	at /usr/local/go/src/runtime/proc.go:404
2  0x0000000001c414b8 in runtime.semacquire1
	at /usr/local/go/src/runtime/sema.go:160
3  0x0000000001c60085 in sync.runtime_Semacquire
	at /usr/local/go/src/runtime/sema.go:62
4  0x0000000001c74b08 in sync.(*WaitGroup).Wait
	at /usr/local/go/src/sync/waitgroup.go:116
5  0x0000000004bd8aa6 in [github.com/pingcap/tidb/pkg/executor.fetchClusterConfig](https://github.com/pingcap/tidb/pkg/executor.fetchClusterConfig)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/memtable_reader.go:273](https://github.com/pingcap/tidb/pkg/executor/memtable_reader.go:273)
6  0x0000000004c12545 in [github.com/pingcap/tidb/pkg/executor.(*ShowExec).fetchShowClusterConfigs](https://github.com/pingcap/tidb/pkg/executor.(*ShowExec).fetchShowClusterConfigs)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/show.go:1425](https://github.com/pingcap/tidb/pkg/executor/show.go:1425)
7  0x0000000004c08bbd in [github.com/pingcap/tidb/pkg/executor.(*ShowExec).fetchAll](https://github.com/pingcap/tidb/pkg/executor.(*ShowExec).fetchAll)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/show.go:183](https://github.com/pingcap/tidb/pkg/executor/show.go:183)
8  0x0000000004c0837a in [github.com/pingcap/tidb/pkg/executor.(*ShowExec).Next](https://github.com/pingcap/tidb/pkg/executor.(*ShowExec).Next)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/show.go:130](https://github.com/pingcap/tidb/pkg/executor/show.go:130)
9  0x000000000430765a in [github.com/pingcap/tidb/pkg/executor/internal/exec.Next](https://github.com/pingcap/tidb/pkg/executor/internal/exec.Next)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:278](https://github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:278)
10  0x0000000004b2697e in [github.com/pingcap/tidb/pkg/executor.(*SelectionExec).Next](https://github.com/pingcap/tidb/pkg/executor.(*SelectionExec).Next)
	at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tidb/pkg/executor/executor.go:1605](https://github.com/pingcap/tidb/pkg/executor/executor.go:1605)