4.x: Additional CQv2 message store optimisations #11112
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
from
b2f11a2 to
66ad60e
Compare
michaelklishin
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
from
66ad60e to
7427cfe
Compare
This comment was marked as outdated.
This comment was marked as outdated.
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
from
4351976 to
84695ff
Compare
|
The first commit greatly improves dirty recovery times. On my machine, with data that's 24 million messages spread over many files and two queues, node recovery goes from 4min30 to less than 2min. The problem was that the old code was gathering messages from queues one by one (meaning 3 or 4 Erlang messages per AMQP message!!). Now it does so per segment file. There are still parts that could be improved for making dirty recovery blazingly fast but they require storing additional state on disk and so will not be investigated fully for now. |
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
2 times, most recently
from
07be784 to
fbf11f5
Compare
lhoguin
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
2 times, most recently
from
fd3a118 to
817c59e
Compare
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
3 times, most recently
from
d8a5536 to
0575002
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
from
0575002 to
fa4e1ad
Compare
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
2 times, most recently
from
0720633 to
001cc83
Compare
This only applies to v2 because modifying this part of the v1 code is seen as too risky considering v1 will soon get removed.
It will always use the ETS index. This change lets us do optimisations that would otherwise not be possible, including 81b2c39. A small functional change is included in this commit: we now always use ets:update_counter to update the ref_count, instead of a mix of update_{counter,fields}. When upgrading to 4.0, the index will be rebuilt for all users that were using a custom index module.
It is no longer needed as only a single module uses it.
essen
force-pushed
the
loic-faster-cq-shared-store-gc
branch
from
001cc83 to
cc73b86
Compare
|
@mkuratczyk did a benchmark and a few checks to see that this PR works as intended. Merging, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Planning to include these for 4.0.