perf(provider): make static-file changeset counts O(files)#23296
Closed
decofe wants to merge 6 commits into
Closed
perf(provider): make static-file changeset counts O(files)#23296decofe wants to merge 6 commits into
decofe wants to merge 6 commits into
Conversation
Co-Authored-By: YK <46377366+yongkangc@users.noreply.github.com>
Contributor
|
Co-authored-by: YK <46377366+yongkangc@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d423c-8436-76a8-8b92-e700bd180b8a
Co-authored-by: YK <46377366+yongkangc@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d423c-8436-76a8-8b92-e700bd180b8a
Co-authored-by: YK <46377366+yongkangc@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d423c-8436-76a8-8b92-e700bd180b8a
Co-authored-by: YK <46377366+yongkangc@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019d423c-8436-76a8-8b92-e700bd180b8a
Co-Authored-By: YK <46377366+yongkangc@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes static-file changeset progress denominator computations by avoiding per-record scans of .csoff sidecar files. Instead, it derives the total count per static file from the last committed changeset-offset record, reducing *_changeset_count() from O(blocks) to O(files) while preserving existing stage semantics.
Changes:
- Replace
account_changeset_count()/storage_changeset_count()implementations to sum per-file totals computed from the last committed.csoffrecord. - Add
ChangesetOffsetReader::total_changes()to compute total changes via a single last-record read. - Add regression tests covering changeset counts across multiple static files for both account and storage changesets.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| crates/storage/provider/src/providers/static_file/manager.rs | Switch changeset count computation to an O(files) approach using a shared helper that reads only the last committed sidecar record per file. |
| crates/static-file/types/src/changeset_offsets.rs | Add total_changes() API on the sidecar reader plus unit tests validating behavior and committed-length handling. |
| crates/storage/provider/src/providers/static_file/mod.rs | Add integration-style tests ensuring account/storage changeset counts are correct across multiple static files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes RETH-694
changeset_count()scans every.csoffrecord to compute a progress-bar denominator for the history indexing stages.Since
.csoffrecords are cumulative[offset, num_changes], the total is justlast.offset + last.num_changes.Before: 502ms (2.2M blocks), ~9s (10.7M blocks)
After: 0.2ms (2.2M blocks), ~0.3ms (10.7M blocks)
Repro and timings: https://gist.github.com/yongkangc/06d1da54a32b4a1818c9432483fef7a6