[v24.3.x] storage: always schedule adjacent segment compaction #24956
+124
−44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI test resultstest results on build#61273
|
The test itself has many steps that make it hard to follow what has happened. Adds some log messages to indicate different steps in the test. (cherry picked from commit 67689fa)
Adds a static constructor to return an optional interval, which per the class comments, should represent an empty interval. I intend on using this for some upcoming bounds checks. (cherry picked from commit 8514bfd)
The methods to find offset range size currently use segment bounds to determine whether a given offset is available to be queried. This doesn't account for the case when the segment set contains offsets that don't fall in the log's offset range (e.g. follow a delete records request that trims mid-segment). This commit adds appropriate bounds checks to both methods. With an upcoming change to merge compact after windowed compaction, test_offset_range_size2_compacted would fail because it would prefix truncate mid-segment following a merge compaction, and then trip over this, hitting an unexpected exception when creating a reader: ``` std::runtime_error: Reader cannot read before start of the log 0 < 887 ``` (cherry picked from commit 69e4666)
We previously fell back on adjacent segment compaction only if there was no new data to compact. In some situations, we've seen the rate of incoming data outpace the compaction interval, causing segments to pile up without ever being merged. This change tweaks the logic to always run adjacent segment compaction after running sliding window compaction. Along the way, a couple tests needed to be tweaked to handle the fact that housekeeping now may merge segments. (cherry picked from commit 08d0433)
andrwng
force-pushed
the
backport-pr-24874-v24.3.x-55
branch
from
48f35d7 to
01d95ed
Compare
|
@andrwng need your review or someone else's review to merge this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #24874
Also pulls in #24880 which is a dependency of this change