Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][broker] Disable system topic message deduplication #22582

Merged
merged 3 commits into from
May 8, 2024

Conversation

mattisonchao
Copy link
Member

@mattisonchao mattisonchao commented Apr 25, 2024

Motivation

Disable deduplication on system topic to avoid recovering deduplication WAL (especially from offloaded topic). Because the system topic usually is a precondition to other topics. Therefore, we should pay attention to topic loading time.

Note: If the system topic loading timeout may cause dependent topics to fail to run.
Dependency diagram: normal topic --rely on--> system topic --rely on--> deduplication recover --may rely on--> (tiered storage)

Modifications

  • MessageDeduplication use Topic#isDeduplicationEnabled to judge if we should enable deduplication.
  • System topic overrides the Topic#isDeduplicationEnabled to return false.

Verifying this change

  • Make sure that the change passes the CI checks.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Apr 25, 2024
@mattisonchao mattisonchao self-assigned this Apr 25, 2024
@mattisonchao mattisonchao added this to the 3.3.0 milestone Apr 25, 2024
Copy link
Member

@dao-jun dao-jun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dao-jun
Copy link
Member

dao-jun commented May 8, 2024

close reopen to trigger ci checks

@dao-jun dao-jun closed this May 8, 2024
@dao-jun dao-jun reopened this May 8, 2024
@codecov-commenter
Copy link

codecov-commenter commented May 8, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 71.62%. Comparing base (bbc6224) to head (53cc8d8).
Report is 235 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #22582      +/-   ##
============================================
- Coverage     73.57%   71.62%   -1.95%     
+ Complexity    32624     4849   -27775     
============================================
  Files          1877     1889      +12     
  Lines        139502   149548   +10046     
  Branches      15299    17591    +2292     
============================================
+ Hits         102638   107117    +4479     
- Misses        28908    33869    +4961     
- Partials       7956     8562     +606     
Flag Coverage Δ
inttests 28.91% <83.33%> (+4.32%) ⬆️
systests 26.04% <33.33%> (+1.71%) ⬆️
unittests 72.10% <83.33%> (-0.75%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...roker/service/persistent/MessageDeduplication.java 81.34% <100.00%> (+0.42%) ⬆️
...sar/broker/service/persistent/PersistentTopic.java 72.14% <100.00%> (-6.32%) ⬇️
.../pulsar/broker/service/persistent/SystemTopic.java 81.25% <100.00%> (+1.25%) ⬆️
...n/java/org/apache/pulsar/broker/service/Topic.java 32.14% <0.00%> (-4.23%) ⬇️

... and 326 files with indirect coverage changes

@dao-jun dao-jun merged commit 5ff0fb9 into apache:master May 8, 2024
96 of 100 checks passed
lhotari pushed a commit that referenced this pull request May 14, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request May 15, 2024
(cherry picked from commit 5ff0fb9)
(cherry picked from commit 34ce38e)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request May 16, 2024
(cherry picked from commit 5ff0fb9)
(cherry picked from commit 34ce38e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants