[PIP-146] ManagedCursorInfo compression #14542
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
doc-required
|
/pulsarbot rerun-failure-checks |
mattisonchao
reviewed
Mar 6, 2022
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
Outdated
Show resolved
Hide resolved
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
Show resolved
Hide resolved
eolivelli
requested changes
Mar 6, 2022
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
Outdated
Show resolved
Hide resolved
nodece
force-pushed
the
managed_cursor_compress
branch
from
30d2021 to
a8c12a2
Compare
nodece
force-pushed
the
managed_cursor_compress
branch
from
fa1013f to
9591d71
Compare
|
@momo-jun a soft reminder: this PR is labeled w/ |
gaoran10
approved these changes
Mar 14, 2022
codelipenghui
approved these changes
Mar 15, 2022
There was a problem hiding this comment.
Great work!
As @Jason918 mentioned in the proposal, to introduce a compression threshold to avoid resources waste for small size cursor data, and it should be also applied to the topic metadata compression, so I think it's better to use a separate PR to add this improvement for both managed ledger metadata and cursor metadata.
|
@eolivelli Please help review this PR again. |
Thanks @codelipenghui, when this PR is merged, I will open a new PR to do this. |
|
@eolivelli Could you review this PR? |
|
@eolivelli Please help review this PR again. |
eolivelli
requested changes
Apr 8, 2022
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
Outdated
Show resolved
Hide resolved
nodece
force-pushed
the
managed_cursor_compress
branch
from
9591d71 to
4cac93e
Compare
|
/pulsarbot rerun-failure-checks |
|
@eolivell your request has been fixed, could you review this again? |
nodece
force-pushed
the
managed_cursor_compress
branch
from
4cac93e to
a609822
Compare
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
Outdated
Show resolved
Hide resolved
|
/pulsarbot rerun-failure-checks |
Signed-off-by: Zixuan Liu <[email protected]>
nodece
force-pushed
the
managed_cursor_compress
branch
from
a609822 to
3b3bc5e
Compare
|
@codelipenghui Thanks, could you review again? |
codelipenghui
approved these changes
Apr 11, 2022
|
@eolivelli Could review this PR again? |
Approved by Enrioc. |
Merged
1 task
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
Fixes apache#14529 ### Motivation The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data. ### Modifications - Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto` for as compression metadata - Add the `managedCursorInfoCompressionType` to `org.apache.pulsar.broker.ServiceConfiguration` and `org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig` - This feature is the same as the implementation of ManagedLedgerInfo compression, so the code is optimized to avoid duplication
Anonymitaet
added
doc-complete
codelipenghui
pushed a commit
to codelipenghui/incubator-pulsar
that referenced
this pull request
Jun 2, 2022
Fixes apache#14529 The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data. - Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto` for as compression metadata - Add the `managedCursorInfoCompressionType` to `org.apache.pulsar.broker.ServiceConfiguration` and `org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig` - This feature is the same as the implementation of ManagedLedgerInfo compression, so the code is optimized to avoid duplication (cherry picked from commit 4398733)
codelipenghui
pushed a commit
that referenced
this pull request
Jun 2, 2022
Fixes #14529 ### Motivation The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data. ### Modifications - Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto` for as compression metadata - Add the `managedCursorInfoCompressionType` to `org.apache.pulsar.broker.ServiceConfiguration` and `org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig` - This feature is the same as the implementation of ManagedLedgerInfo compression, so the code is optimized to avoid duplication (cherry picked from commit 4398733)
codelipenghui
added
release/2.9.3
release/2.10.1
cherry-picked/branch-2.9
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 6, 2022
Fixes apache#14529 ### Motivation The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data. ### Modifications - Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto` for as compression metadata - Add the `managedCursorInfoCompressionType` to `org.apache.pulsar.broker.ServiceConfiguration` and `org.apache.bookkeeper.mledger.ManagedLedgerFactoryConfig` - This feature is the same as the implementation of ManagedLedgerInfo compression, so the code is optimized to avoid duplication (cherry picked from commit 4398733) (cherry picked from commit 70c7794)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Zixuan Liu [email protected]
Fixes #14529
Motivation
The cursor data is managed by ZooKeeper/etcd metadata store. When cursor data becomes more and more, the data size will increase and will take a lot of time to pull the data. Therefore, it is necessary to add compression for the cursor, which can reduce the size of data and reduce the time of pulling data.
Modifications
ManagedCursorInfoMetadatamessage toMLDataFormats.protofor as compression metadatamanagedCursorInfoCompressionTypetoorg.apache.pulsar.broker.ServiceConfigurationandorg.apache.bookkeeper.mledger.ManagedLedgerFactoryConfigManagedLedgerInfo compress
Documentation
doc-requiredAdd
managedCursorInfoCompressionTypeto broker configuration