ingest: retry failed regions when batch scatter regions by GMHDBJD · Pull Request #61722 · pingcap/tidb

GMHDBJD · 2025-06-13T07:24:01Z

What problem does this PR solve?

Issue Number: close #60077

Problem Summary:

What changed and how does it work?

retry failed regions when batch scatter regions
cherrypick lightning: added retry times and two new retry reasons | tidb-test=release-8.1.2 #60839

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-06-13T07:24:20Z

Hi @GMHDBJD. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-06-13T07:40:46Z

Codecov Report

Attention: Patch coverage is 84.61538% with 6 lines in your changes missing coverage. Please review.

Project coverage is 75.8085%. Comparing base (f770560) to head (0a83198).
Report is 156 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61722        +/-   ##
================================================
+ Coverage   73.1149%   75.8085%   +2.6935%     
================================================
  Files          1730       1782        +52     
  Lines        481167     500011     +18844     
================================================
+ Hits         351805     379051     +27246     
+ Misses       107842      97946      -9896     
- Partials      21520      23014      +1494

Flag	Coverage Δ
integration	`49.5291% <47.3684%> (?)`
unit	`73.1016% <84.6153%> (+0.7420%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`64.6215% <84.6153%> (+17.6387%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

GMHDBJD · 2025-06-16T09:17:30Z

/retest

tiprow · 2025-06-16T09:17:52Z

@GMHDBJD: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

DEPS.bzl

Benjamin2037

LGTM

br/pkg/restore/split/client.go

ti-chi-bot · 2025-06-20T03:41:01Z

[LGTM Timeline notifier]

Timeline:

2025-06-19 09:52:59.332993437 +0000 UTC m=+351832.056172418: ☑️ agreed by Benjamin2037.
2025-06-20 03:41:00.226545986 +0000 UTC m=+415912.949724992: ☑️ agreed by tangenta.

bb7133

LGTM

ti-chi-bot · 2025-06-23T02:34:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, Benjamin2037, Leavrth, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Benjamin2037,Leavrth,bb7133,tangenta]
~~br/OWNERS~~ [Leavrth]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

3pointer · 2025-06-23T03:48:48Z

br/pkg/restore/split/client.go

+					failedRegions = append(failedRegions, region)
+				}
+			}
+			newRegions = failedRegions


I’m concerned that the failure-handling logic here may not be robust. If a region is mistakenly judged as needing to be scattered and the operation fails, it appears that this region won’t be retried later.

Consider this scenario: a small number of regions fail to add scatter operators, while others succeed. However, due to the asynchronous nature of operator execution, even regions that successfully had operators added might still time out or encounter other issues afterward. Does this scenario align with the intended design?

Benjamin2037 · 2025-06-23T05:03:36Z

/retest

* origin/master: (129 commits) domain: Fix the issue that the min start ts doesn't correctly block keyspace-level GC (pingcap#61925) br: better control pd scheduler pause during log restore with filter (pingcap#61819) session: rename circuit breaker sysvar (pingcap#61951) dxfservice: create store for SYSTEM keyspace (pingcap#61752) docs: fix a dead link in CONTRIBUTORS.md (pingcap#61923) metrics/nextgengrafana: display keyspace separately (pingcap#61823) lightning: fix length check may be skipped for first line (pingcap#61874) planner: support `explain [analyze] <plan_digest>` for `explain explore` (pingcap#61942) planner: record explored plans into `tidb_statement_stats` when running `explain explore` (pingcap#61850) fix(runaway): ensure DistSQLContext's checker is synchronized with session variables (pingcap#61907) expression,planner: reuse the propOuterJoinConstSolver to improve performance (pingcap#61913) ddl,planner: remove unused and meaningless code (pingcap#61936) planner: remove unused field from physicalTableScan. (pingcap#61935) workload-learning: Extract metrics from cluster statements stats (pingcap#61378) executor: minor cleanup in builder.go (pingcap#61924) session: rename GetDomainInfoSchema to GetLatestInfoSchema (pingcap#61894) ingest: retry failed regions when batch scatter regions (pingcap#61722) planner: add tpch q1,q2,q3 benchmark (pingcap#61898) planner: fix uninit timeout for loading bindings (pingcap#61891) executor: report error when admin check on multiple tables (pingcap#61828) ...

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-07-08T07:12:24Z

In response to a cherrypick label: new pull request created to branch release-8.1: #62287.
But this PR has conflicts, please resolve them!

ti-chi-bot · 2025-07-08T07:12:27Z

In response to a cherrypick label: new pull request created to branch release-8.5: #62288.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

) close #60077

… (pingcap#62288) close pingcap#60077

…cap#61722) (pingcap#62288)" This reverts commit b5c7e2f.

retry failed regions when batch scatter regions

351949f

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 13, 2025

pick pingcap#60839

bd384a7

add ut

0a83198

tangenta reviewed Jun 17, 2025

View reviewed changes

DEPS.bzl Show resolved Hide resolved

Benjamin2037 approved these changes Jun 19, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 19, 2025

tangenta reviewed Jun 20, 2025

View reviewed changes

br/pkg/restore/split/client.go Show resolved Hide resolved

tangenta approved these changes Jun 20, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 20, 2025

Leavrth approved these changes Jun 20, 2025

View reviewed changes

bb7133 approved these changes Jun 23, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jun 23, 2025

3pointer reviewed Jun 23, 2025

View reviewed changes

ti-chi-bot bot merged commit c6cf760 into pingcap:master Jun 23, 2025
41 of 46 checks passed

ti-chi-bot bot added needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. labels Jul 8, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 8, 2025

This is an automated cherry-pick of pingcap#61722

d76f485

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 8, 2025

ingest: retry failed regions when batch scatter regions (#61722) #62287

Closed

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 8, 2025

This is an automated cherry-pick of pingcap#61722

7099436

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jul 8, 2025

ingest: retry failed regions when batch scatter regions (#61722) #62288

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jul 16, 2025

ingest: retry failed regions when batch scatter regions (#61722) (#62288

62f5427

) close #60077

JQWong7 pushed a commit to stevenayu/tidb that referenced this pull request Jul 30, 2025

ingest: retry failed regions when batch scatter regions (pingcap#61722)…

b5c7e2f

… (pingcap#62288) close pingcap#60077

JQWong7 added a commit to stevenayu/tidb that referenced this pull request Jul 30, 2025

Revert "ingest: retry failed regions when batch scatter regions (ping…

87d416b

…cap#61722) (pingcap#62288)" This reverts commit b5c7e2f.

Conversation

GMHDBJD commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Jun 13, 2025

Uh oh!

codecov bot commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GMHDBJD commented Jun 16, 2025

Uh oh!

tiprow bot commented Jun 16, 2025

Uh oh!

Uh oh!

Benjamin2037 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ti-chi-bot bot commented Jun 20, 2025

[LGTM Timeline notifier]

Uh oh!

bb7133 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

Uh oh!

3pointer Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Benjamin2037 commented Jun 23, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Jul 8, 2025

Uh oh!

ti-chi-bot commented Jul 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

GMHDBJD commented Jun 13, 2025 •

edited

Loading

codecov bot commented Jun 13, 2025 •

edited

Loading