Migrate LogIndex block numbers from int to uint#10430
Closed
LukaszRozmej wants to merge 5 commits intofeature/log-index-2from
Closed
Migrate LogIndex block numbers from int to uint#10430LukaszRozmej wants to merge 5 commits intofeature/log-index-2from
LukaszRozmej wants to merge 5 commits intofeature/log-index-2from
Conversation
Use key-based IsCompressed detection (transient vs finalized postfix) instead of sign-bit trick that breaks for block numbers >= 2^31. Switch CompressionAlgorithm delegates from int to uint signatures, removing MemoryMarshal.Cast<uint, int> workarounds. Replace unsafe BinarySearch cast with proper uint binary search implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LogIndexHighBlockNumberTests verifying that block numbers crossing the int.MaxValue boundary are correctly stored, compressed, and retrieved. Fix collision comment to use 100ms/block timing (~13.6 years). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
Author
|
Can't make it stable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #8464
Changes
inttouint, supporting block numbers up to ~4.3 billionIsCompresseddetection: replace the sign-bit trick (len > 0on a negated int32 marker) with key-based detection — transient keys (BackwardMerge/ForwardMergepostfixes) are never compressed, finalized keys are always compressed. The old approach broke for block numbers >= 2^31 because the compression marker (stored as a negative int) could collide with valid uint block numbersBinarySearch: replaceMemoryMarshal.Cast<uint, int>+ built-inBinarySearchwith a proper uint binary search, since the cast produces incorrect ordering for values >= 2^31CompressionAlgorithmdelegates fromReadOnlySpan<int>/Span<int>toReadOnlySpan<uint>/Span<uint>, removingMemoryMarshal.Cast<uint, int>workarounds in compress/decompress pathsTestFixtureDataentry withstartNum: int.MaxValue - 200to the integration test suite, exercising block numbers crossing the 2^31 boundary across all existing test methodsHow compression detection works now
Each filter (address/topic) maps to multiple DB keys. Two are transient (mutable, uncompressed):
filter || 0x00000000filter || 0xFFFFFFFFAny number of finalized keys store immutable, compressed data:
filter || (first_block + 1)IsCompressedchecks whether the key ends with theBackwardMergeorForwardMergepostfix. If it does, the value is uncompressed raw uint sequences. Otherwise it is a finalized compressed value whose first 4 bytes encode the decompressed element count.Postfix collisions (finalized key matching a transient postfix) occur at block
0xFFFFFFFFand0xFFFFFFFE— roughly 1600 years away at 12s/block.Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
The new
TestFixtureData(5, 100, startNum: (uint)int.MaxValue - 200)creates 500 blocks spanning 2147483447-2147483946, crossingint.MaxValueat block 2147483647. All existing integration test methods (set/get, backwards sync, reorg, compaction, concurrent access, multi-instance, failure recovery) run automatically against this fixture.Documentation
Requires documentation update
Requires explanation in Release Notes