domain: Fix the issue that the min start ts doesn't correctly block keyspace-level GC by MyonKeminta · Pull Request #61925 · pingcap/tidb

MyonKeminta · 2025-06-23T05:42:58Z

What problem does this PR solve?

Issue Number: ref #58720

Problem Summary:

What changed and how does it work?

TiDB saves the min start ts to etcd, but the path to store this information is always without the keyspace prefix. In keyspace-level GC mode, this causes the min start ts not visible to GC and unable to block GC from advancing.
In our final refactor plan, the min start ts mechanism is expected to be replaced by the GC barriers, but it still needs time to get implemented. We need a temporary quick fix for this problem before the replacement is ready.

Manually testes:

Running a transaction in keyspace 1 for 30 minutes, not blocking other keyspaces:

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

…eyspace-level GC Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot · 2025-06-23T05:43:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tiprow · 2025-06-23T05:43:20Z

Hi @MyonKeminta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

MyonKeminta · 2025-06-23T05:44:30Z

/test all

tiprow · 2025-06-23T05:44:51Z

@MyonKeminta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-06-23T06:01:42Z

Codecov Report

Attention: Patch coverage is 78.94737% with 4 lines in your changes missing coverage. Please review.

Project coverage is 75.1877%. Comparing base (465b166) to head (9c77b5e).
Report is 28 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61925        +/-   ##
================================================
+ Coverage   73.0213%   75.1877%   +2.1663%     
================================================
  Files          1735       1781        +46     
  Lines        481288     496500     +15212     
================================================
+ Hits         351443     373307     +21864     
+ Misses       108298     100076      -8222     
- Partials      21547      23117      +1570

Flag	Coverage Δ
integration	`49.1821% <57.8947%> (?)`
unit	`72.4408% <78.9473%> (+0.1710%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.9086% <ø> (+17.3357%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta · 2025-06-23T07:46:10Z

/test all

tiprow · 2025-06-23T07:46:32Z

@MyonKeminta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

MyonKeminta · 2025-06-23T08:37:05Z

/test all

tiprow · 2025-06-23T08:37:27Z

@MyonKeminta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta · 2025-06-24T05:55:20Z

/retest

tiprow · 2025-06-24T05:55:43Z

@MyonKeminta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot · 2025-06-24T06:53:43Z

[LGTM Timeline notifier]

Timeline:

2025-06-24 06:32:13.996579665 +0000 UTC m=+771786.719758647: ☑️ agreed by cfzjywxk.
2025-06-24 06:53:42.798890012 +0000 UTC m=+773075.522068994: ☑️ agreed by tiancaiamao.

MyonKeminta · 2025-06-24T06:55:34Z

/retest

tiprow · 2025-06-24T06:55:57Z

@MyonKeminta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-06-24T06:56:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, tangenta, tiancaiamao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cfzjywxk,tangenta,tiancaiamao]
~~pkg/domain/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cfzjywxk · 2025-06-24T09:48:15Z

/test pull-lightning-integration-test

tiprow · 2025-06-24T09:48:37Z

@cfzjywxk: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

cfzjywxk · 2025-06-24T09:50:46Z

/ok-to-test

cfzjywxk · 2025-06-24T11:15:24Z

/retest

* origin/master: (129 commits) domain: Fix the issue that the min start ts doesn't correctly block keyspace-level GC (pingcap#61925) br: better control pd scheduler pause during log restore with filter (pingcap#61819) session: rename circuit breaker sysvar (pingcap#61951) dxfservice: create store for SYSTEM keyspace (pingcap#61752) docs: fix a dead link in CONTRIBUTORS.md (pingcap#61923) metrics/nextgengrafana: display keyspace separately (pingcap#61823) lightning: fix length check may be skipped for first line (pingcap#61874) planner: support `explain [analyze] <plan_digest>` for `explain explore` (pingcap#61942) planner: record explored plans into `tidb_statement_stats` when running `explain explore` (pingcap#61850) fix(runaway): ensure DistSQLContext's checker is synchronized with session variables (pingcap#61907) expression,planner: reuse the propOuterJoinConstSolver to improve performance (pingcap#61913) ddl,planner: remove unused and meaningless code (pingcap#61936) planner: remove unused field from physicalTableScan. (pingcap#61935) workload-learning: Extract metrics from cluster statements stats (pingcap#61378) executor: minor cleanup in builder.go (pingcap#61924) session: rename GetDomainInfoSchema to GetLatestInfoSchema (pingcap#61894) ingest: retry failed regions when batch scatter regions (pingcap#61722) planner: add tpch q1,q2,q3 benchmark (pingcap#61898) planner: fix uninit timeout for loading bindings (pingcap#61891) executor: report error when admin check on multiple tables (pingcap#61828) ...

domain: Fix the issue that the min start ts doesn't correctly block k…

8b362de

…eyspace-level GC Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 23, 2025

MyonKeminta mentioned this pull request Jun 23, 2025

Refactor GC related PD APIs and GC procedures #58720

Open

8 tasks

Allow nil tikvCodec

c5802e4

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 23, 2025

update pd client to make the helper function available

bba3bc8

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta marked this pull request as ready for review June 24, 2025 05:04

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2025

Fix bazel

9c77b5e

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

cfzjywxk requested a review from tiancaiamao June 24, 2025 06:30

cfzjywxk approved these changes Jun 24, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 24, 2025

tiancaiamao approved these changes Jun 24, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 24, 2025

tangenta approved these changes Jun 24, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jun 24, 2025

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Jun 24, 2025

ti-chi-bot bot merged commit 22fe893 into pingcap:master Jun 24, 2025
26 checks passed

MyonKeminta deleted the m/save-min-start-ts-to-etcd-considering-keyspace branch June 25, 2025 04:02

owlsome2501 mentioned this pull request Jun 26, 2025

There is a 14.5% performance regression in sysbench after PR#61925 #62022

Closed

tiancaiamao mentioned this pull request Jul 7, 2025

GC is not blocked when query is running #61011

Closed

Conversation

MyonKeminta commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

Uh oh!

tiprow bot commented Jun 23, 2025

Uh oh!

MyonKeminta commented Jun 23, 2025

Uh oh!

tiprow bot commented Jun 23, 2025

Uh oh!

codecov bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MyonKeminta commented Jun 23, 2025

Uh oh!

tiprow bot commented Jun 23, 2025

Uh oh!

MyonKeminta commented Jun 23, 2025

Uh oh!

tiprow bot commented Jun 23, 2025

Uh oh!

MyonKeminta commented Jun 24, 2025

Uh oh!

tiprow bot commented Jun 24, 2025

Uh oh!

ti-chi-bot bot commented Jun 24, 2025

[LGTM Timeline notifier]

Uh oh!

MyonKeminta commented Jun 24, 2025

Uh oh!

tiprow bot commented Jun 24, 2025

Uh oh!

ti-chi-bot bot commented Jun 24, 2025

Uh oh!

cfzjywxk commented Jun 24, 2025

Uh oh!

tiprow bot commented Jun 24, 2025

Uh oh!

cfzjywxk commented Jun 24, 2025

Uh oh!

cfzjywxk commented Jun 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MyonKeminta commented Jun 23, 2025 •

edited

Loading

codecov bot commented Jun 23, 2025 •

edited

Loading