Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve] [broker] Avoid repeated Read-and-discard when using Key_Shared mode #22245

Merged
merged 12 commits into from
Mar 29, 2024
Merged

[improve] [broker] Avoid repeated Read-and-discard when using Key_Shared mode #22245

merged 12 commits into from
Mar 29, 2024

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Mar 11, 2024 •
edited
Loading

Motivation

When using Key_Shared subscription, the broker may discard a part of the result reading from BK to avoid breaking the order of consumption, but this may cause repeated Read-and-discard, which wastes a lot of BK traffic.

  • Read entries from the Replay queue.
  • The entries can not be sent to the consumer due to they are larger than consumer.maxReadPosition
  • Nothing was delivered to the client, but the Broker did more and more BK readings.

You can reproduce the issue by testNoRepeatedReadAndDiscard

Modifications

Before reading the BK, filter out the positions that will be discarded.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 11, 2024
@poorbarcode poorbarcode self-assigned this Mar 11, 2024
@poorbarcode poorbarcode added this to the 3.3.0 milestone Mar 11, 2024
@poorbarcode poorbarcode reopened this Mar 11, 2024
lhotari
lhotari reviewed Mar 12, 2024
View reviewed changes
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the KeySharedSubscriptionTest failures that are in the flaky test suite.

@poorbarcode
Copy link
Contributor Author

@lhotari

Please check the KeySharedSubscriptionTest failures in the flaky test suite.

Fixed, thanks

@poorbarcode poorbarcode requested a review from lhotari March 12, 2024 13:25
@poorbarcode poorbarcode force-pushed the fix/repeat_read_in_key_shared_mode branch from 4f5a284 to 326c0cc Compare March 25, 2024 15:00
@poorbarcode
Copy link
Contributor Author

rebase master

@poorbarcode poorbarcode force-pushed the fix/repeat_read_in_key_shared_mode branch from 326c0cc to ea66ce1 Compare March 28, 2024 06:55
@poorbarcode
Copy link
Contributor Author

rebase master

@poorbarcode poorbarcode force-pushed the fix/repeat_read_in_key_shared_mode branch from d887253 to 7fb0a89 Compare March 28, 2024 19:55
@codelipenghui codelipenghui merged commit e34ea62 into apache:master Mar 29, 2024
49 of 50 checks passed
poorbarcode added a commit that referenced this pull request Mar 29, 2024
@poorbarcode
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
5b37e84 
…red mode (#22245)

(cherry picked from commit e34ea62)
poorbarcode added a commit that referenced this pull request Mar 29, 2024
@poorbarcode
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
03493ef 
…red mode (#22245)

(cherry picked from commit e34ea62)
poorbarcode added a commit that referenced this pull request Mar 29, 2024
@poorbarcode
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
55568bf 
…red mode (#22245)
poorbarcode added a commit that referenced this pull request Mar 29, 2024
@poorbarcode
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
726de46 
…red mode (#22245)

(cherry picked from commit e34ea62)
Technoboy- pushed a commit to Technoboy-/pulsar that referenced this pull request Apr 1, 2024
@poorbarcode @Technoboy-
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
5ef60ed 
…red mode (apache#22245)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 15, 2024
@poorbarcode @mukesh-ctds
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
2ff9e98 
…red mode (apache#22245)

(cherry picked from commit e34ea62)
(cherry picked from commit 5b37e84)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 17, 2024
@poorbarcode @mukesh-ctds
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
0c7a581 
…red mode (apache#22245)

(cherry picked from commit e34ea62)
(cherry picked from commit 5b37e84)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 17, 2024
@poorbarcode @mukesh-ctds
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
9484ba1 
…red mode (apache#22245)

(cherry picked from commit e34ea62)
(cherry picked from commit 5b37e84)
@poorbarcode poorbarcode mentioned this pull request Apr 18, 2024
Merged
4 tasks
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 19, 2024
@poorbarcode @mukesh-ctds
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
3bd3ce0 
…red mode (apache#22245)

(cherry picked from commit e34ea62)
(cherry picked from commit 5b37e84)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Apr 23, 2024
@poorbarcode @srinath-ctds
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
3eb5c2d 
…red mode (apache#22245)

(cherry picked from commit e34ea62)
(cherry picked from commit 5b37e84)
nodece pushed a commit to ascentstream/pulsar that referenced this pull request May 13, 2024
@poorbarcode @nodece
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
1fcad34 
…red mode (apache#22245)
@lhotari lhotari mentioned this pull request Aug 20, 2024
Closed
2 tasks
@lhotari lhotari mentioned this pull request Sep 11, 2024
Merged
4 tasks
@lhotari lhotari mentioned this pull request Oct 25, 2024
Open
3 tasks
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
@poorbarcode @hanmz
[improve] [broker] Avoid repeated Read-and-discard when using Key_Sha…
ff49025 
…red mode (apache#22245)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@codelipenghui codelipenghui codelipenghui approved these changes

@lhotari lhotari Awaiting requested review from lhotari

Assignees

@poorbarcode poorbarcode

Labels
cherry-picked/branch-2.10 cherry-picked/branch-2.11 cherry-picked/branch-3.0 cherry-picked/branch-3.2 doc-not-needed Your PR changes do not impact docs ready-to-test release/2.10.7 release/2.11.5 release/3.0.5 release/3.2.3
Projects
None yet
Milestone
3.3.0
Development

Successfully merging this pull request may close these issues.
3 participants
@poorbarcode @lhotari @codelipenghui