[Branch-2.7][Cherry-pick] Fix reader skipped remaining compacted data during the topic unloading #16300
+39
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick #13629
Motivation
To fix the reader skipping remaining compacted data while the topic has been unloaded.
#11287 fixed the data skipped issue while the reader first time to read the messages
with the earliest position. But if the reader has consumed some messages from the
compacted ledger but not all, the start position will not be
earliest
, the brokerwill rewind the cursor for the reader to the next valid position of the original topic.
So the remaining messages in the compacted ledger will be skipped.
Here are the logs from the broker:
There some many compacted messages after
181759:13
, but the broker will not dispatch them to the reader.The issue also can be reproduced by the unit test that was added in this PR.
Modification
If the cursor with
readCompacted = true
, just rewind to the next message of the mark delete position,so that the reader can continue to read the data from the compacted ledger.
Verification
A new test added for testing the reader can get all the compacted messages and non-compacted messages from the topic during the topic unloading.
Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
doc-required
(If you need help on updating docs, create a doc issue)
no-need-doc
(Please explain why)
doc
(If this PR contains doc changes)