Skip to content

[close #348] support filter for key & value #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Aug 25, 2023
Merged

Conversation

pingyu
Copy link
Collaborator

@pingyu pingyu commented Aug 17, 2023

What problem does this PR solve?

Issue Number: close #348

Problem Description: Support filter for key or value

What is changed and how does it work?

  • Add kv_filter to filter event match the pattern provided by user.

Add metric "filter-out" (the test case is that all events were filtered out):
image

Code changes

Check List for Tests

This PR has been tested by at least one of the following methods:

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Side effects

  • No side effects

Related changes

  • Need to update the documentation

To reviewers

Please follow these principles to check this pull requests:

  • Concentration. One pull request should only do one thing. No matter how small it is, the change does exactly one thing and gets it right. Don't mix other changes into it.
  • Tests. A pull request should be test covered, whether the tests are unit tests, integration tests, or end-to-end tests. Tests should be sufficient, correct and don't slow down the CI pipeline largely.
  • Functionality. The pull request should implement what the author intends to do and fit well in the existing code base, resolve a real problem for TiDB/TiKV users. To get the author's intention and the pull request design, you could follow the discussions in the corresponding GitHub issue or internal.tidb.io topic.
  • Style. Code in the pull request should follow common programming style. (For Go, we have style checkers in CI). However, sometimes the existing code is inconsistent with the style guide, you should maintain consistency with the existing code or file a new issue to fix the existing code style first.
  • Documentation. If a pull request changes how users build, test, interact with, or release code, you must check whether it also updates the related documentation such as READMEs and any generated reference docs. Similarly, if a pull request deletes or deprecates code, you must check whether or not the corresponding documentation should also be deleted.
  • Performance. If you find the pull request may affect performance, you could ask the author to provide a benchmark result.

(The above text mainly refers to TiDB Development Guide. It's also highly recommended to read about Writing code review comments)

@pingyu pingyu marked this pull request as draft August 17, 2023 13:30
pingyu added 6 commits August 22, 2023 14:37
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
@codecov
Copy link

codecov bot commented Aug 24, 2023

Codecov Report

Merging #349 (c305708) into main (c340278) will decrease coverage by 0.1262%.
The diff coverage is 63.7500%.

Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##               main       #349        +/-   ##
================================================
- Coverage   61.1272%   61.0011%   -0.1262%     
================================================
  Files           240        240                
  Lines         20261      20357        +96     
================================================
+ Hits          12385      12418        +33     
- Misses         6758       6822        +64     
+ Partials       1118       1117         -1     
Flag Coverage Δ *Carryforward flag
br 60.6802% <ø> (ø) Carriedforward from 5e65388
cdc 61.1485% <63.7500%> (-0.1856%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

Files Changed Coverage Δ
cdc/cdc/kv/metrics.go 0.0000% <0.0000%> (ø)
cdc/cdc/processor/pipeline/keyspan.go 0.0000% <0.0000%> (ø)
cdc/cdc/processor/pipeline/puller.go 0.0000% <0.0000%> (ø)
cdc/pkg/config/replica_config.go 63.2653% <0.0000%> (-5.6236%) ⬇️
cdc/pkg/util/ctx.go 76.4705% <0.0000%> (-12.1658%) ⬇️
cdc/cdc/kv/region_worker.go 78.6885% <33.3333%> (-0.8949%) ⬇️
cdc/pkg/cmd/cli/cli_changefeed_update.go 28.2051% <66.6666%> (+2.9877%) ⬆️
cdc/pkg/util/kv_filter.go 91.6666% <91.6666%> (ø)
cdc/cdc/kv/client.go 83.8848% <100.0000%> (-2.1382%) ⬇️

... and 17 files with indirect coverage changes

@pingyu
Copy link
Collaborator Author

pingyu commented Aug 24, 2023

/run-integration-test

@pingyu pingyu marked this pull request as ready for review August 24, 2023 08:29
@pingyu
Copy link
Collaborator Author

pingyu commented Aug 24, 2023

/run-pull-integration-test

Signed-off-by: Ping Yu <[email protected]>
@purelind
Copy link

/test ?

@purelind
Copy link

/run-integration-test

1 similar comment
@pingyu
Copy link
Collaborator Author

pingyu commented Aug 24, 2023

/run-integration-test

pingyu added 2 commits August 24, 2023 22:05
Signed-off-by: Ping Yu <[email protected]>
Copy link
Contributor

@zeminzhou zeminzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@zeminzhou
Copy link
Contributor

/run-integration-test

@pingyu pingyu merged commit 9b4dc22 into tikv:main Aug 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cdc: Support filter for key or value
3 participants