Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: support tidb_redact_log for explain #54553

Merged
merged 4 commits into from
Jul 23, 2024
Merged

Conversation

hawkingrei
Copy link
Member

@hawkingrei hawkingrei commented Jul 10, 2024

What problem does this PR solve?

Issue Number: close #54565

Problem Summary:

What changed and how does it work?

The main issue is that TiDB's explain function does not support tidb_redact_log, which has caused this problem to occur.

Due to the substantial size of this PR, I will outline the modifications made and the tests conducted.

The changement of StringCtx

StringCtx is used to generate information in the explain function and is also utilized in log generation. Therefore, it cannot directly read ctx to determine the redact mode.

So the changement of expression package is to add the redact parameter into StringCtx and force using the redactDisable mode when to raise the error.

so I refactor the StringWithCtx's implementation of Constant,Column. but planner/core.ToString/StringifyExpressionsWithCtx is to print the debug info for testing. so it forces to use the redactDisable

pkg/util/ranger/types.go is to support the redact for the ranger.

the changement of the planner

The main focus is on the physical operator to resolve the explain output. The primary modifications include enabling the expression to support redaction by passing the redact parameter into the expression's explain method. Additionally, some parameters of the operator itself are subjected to redaction.

the changement of the ranger

Enable the range to support redaction mode.

the main test cases

  • pkg/planner/core/tests/redact
  • pkg/planner/core/plan_cache_test.go
    - avoid plan cache polluting by the redact.

Check List

Tests

  • Unit test
  1. I add some test cases to make redact results right
  2. To prevent redaction from affecting the plan cache and plan digest tests, specific tests were conducted for different redaction scenarios to observe the plan cache hit rate.
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 10, 2024
Copy link

codecov bot commented Jul 10, 2024

Codecov Report

Attention: Patch coverage is 79.61538% with 53 lines in your changes missing coverage. Please review.

Project coverage is 56.0782%. Comparing base (504960d) to head (e749c1c).
Report is 16 commits behind head on master.

Additional details and impacted files
@@                Coverage Diff                @@
##             master     #54553         +/-   ##
=================================================
- Coverage   72.7518%   56.0782%   -16.6736%     
=================================================
  Files          1555       1680        +125     
  Lines        437652     615001     +177349     
=================================================
+ Hits         318400     344882      +26482     
- Misses        99655     246673     +147018     
- Partials      19597      23446       +3849     
Flag Coverage Δ
integration 37.4088% <46.9230%> (?)
unit 71.7433% <71.7054%> (-0.0343%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9656% <ø> (ø)
parser ∅ <ø> (∅)
br 52.9302% <ø> (+7.0589%) ⬆️

@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 10, 2024
@hawkingrei hawkingrei marked this pull request as draft July 11, 2024 06:38
@ti-chi-bot ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. and removed do-not-merge/needs-linked-issue labels Jul 11, 2024
@hawkingrei hawkingrei marked this pull request as ready for review July 11, 2024 07:18
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2024
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-tests-checked size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 11, 2024
@hawkingrei
Copy link
Member Author

/ok-to-test

@ti-chi-bot ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Jul 11, 2024
@hawkingrei
Copy link
Member Author

/retest

1 similar comment
@hawkingrei
Copy link
Member Author

/retest

@hawkingrei hawkingrei force-pushed the support branch 5 times, most recently from c8a1e5a to d396baa Compare July 15, 2024 11:42
@hawkingrei
Copy link
Member Author

/retest

Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@hawkingrei
Copy link
Member Author

/retest

Signed-off-by: Weizhen Wang <[email protected]>
@hawkingrei hawkingrei requested a review from qw4990 July 22, 2024 06:09
@ti-chi-bot ti-chi-bot bot added the approved label Jul 22, 2024
@hawkingrei
Copy link
Member Author

/hold

@ti-chi-bot ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 22, 2024
@hawkingrei
Copy link
Member Author

@qw4990 PTAL, Please confirm that it will not affect the plan cache.

@hawkingrei hawkingrei requested a review from winoros July 23, 2024 03:01
Copy link

ti-chi-bot bot commented Jul 23, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, lance6716, qw4990, tangenta, windtalker, xhebox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hawkingrei
Copy link
Member Author

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2024
@hawkingrei
Copy link
Member Author

/retest

@ti-chi-bot ti-chi-bot bot merged commit f5ac1c4 into pingcap:master Jul 23, 2024
22 of 23 checks passed
@hawkingrei
Copy link
Member Author

/cherrypick release-8.1
/cherrypick release-7.5

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Aug 8, 2024
@ti-chi-bot
Copy link
Member

@hawkingrei: new pull request created to branch release-7.5: #55307.

In response to this:

/cherrypick release-8.1
/cherrypick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Aug 8, 2024
@ti-chi-bot
Copy link
Member

@hawkingrei: new pull request created to branch release-8.1: #55308.

In response to this:

/cherrypick release-8.1
/cherrypick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@hawkingrei hawkingrei deleted the support branch August 8, 2024 09:31
ti-chi-bot bot pushed a commit that referenced this pull request Aug 22, 2024
hawkingrei pushed a commit to ti-chi-bot/tidb that referenced this pull request Sep 23, 2024
hawkingrei added a commit to ti-chi-bot/tidb that referenced this pull request Sep 24, 2024
ti-chi-bot bot pushed a commit that referenced this pull request Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm ok-to-test Indicates a PR is ready to be tested. release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The tidb_redact_log cannot be applied to the explain.
8 participants