tidb lightning: daily run find 'Lightning is stuck" for hours

[shaoxiqian]

[git-hash=b9a31b231a7d9a64da81cb071b3db26fcb55cc38]

toml

        [lightning]
        level = "info"
        check-requirements = false
        status-addr = ':8289'
        file = "/tmp/tidb-lightning_1719543717.log"

        [tikv-importer]
        backend = "local"
        incremental-import = false
        sorted-kv-dir = "/tiup/sorted-kv-dir"

        [tidb]
        # Information of the target cluster
        port = 4000
        user = "root"
        password = ""
        host = "tidb-1-peer"
        status-port = 10080
        pd-addr = "pd-peer:2379"

        [mydumper]
        no-schema = true
        data-source-dir = 's3://tmp/test?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio-peer:9000'
        [mydumper.csv]
        header = false

        [checkpoint]
        # Whether to enable checkpoints.
        enable = true
        driver = "file"

        [post-restore]
        checksum = true

[kennytm]

According to the goroutine we are stuck at

tidb/br/pkg/restore/split/split.go

Lines 96 to 109 in b9a31b2

	func PaginateScanRegion(
	ctx context.Context, client SplitClient, startKey, endKey []byte, limit int,
	) ([]*RegionInfo, error) {
	if len(endKey) != 0 && bytes.Compare(startKey, endKey) > 0 {
	return nil, errors.Annotatef(berrors.ErrInvalidRange, "startKey > endKey, startKey: %s, endkey: %s",
	hex.EncodeToString(startKey), hex.EncodeToString(endKey))
	}
	var (
	lastRegions []*RegionInfo
	err error
	backoffer = NewWaitRegionOnlineBackoffer()
	)
	_ = utils.WithRetry(ctx, func() error {

which is busy waiting on

tidb/br/pkg/utils/retry.go

Lines 239 to 244 in b9a31b2

	select {
	case <-ctx.Done():
	// allErrors must not be `nil` here, so ignore the context error.
	return *new(T), allErrors
	case <-time.After(backoffer.NextBackoff(err)):
	}

there are no error logs, and the select does not seem to be waiting for more than 1 minute.

I suspect this condition is erroneously triggered

tidb/br/pkg/restore/split/split.go

Lines 132 to 136 in b9a31b2

	// if the number of regions changed, we can infer TiKV side really
	// made some progress so don't increase the retry times.
	if len(regions) != len(lastRegions) {
	backoffer.Stat.ReduceRetry()
	}

which put the retry loop into an infinite loop

[lance6716]

caused by PD updates the API in #54153 and test cluster is using old version PD. Lightning will not print the error details and this needs to be improved.

Still in discuss about what's the expected fix