feat(journaling)!: stream catalog enumeration - #11211
Merged
ReubenBond merged 6 commits intoSep 9, 2026
Merged
Conversation
Add an optional paging contract and an indexed volatile implementation with prefix-scoped, instance-local continuation tokens. Preserve sorted ListAsync enumeration and storage existence ordering. Adapt the catalog API and volatile index work authored by Reuben Bond in e9dba90 during the review of KSemenenko's advanced reminder proposal. Native provider adaptations and conformance coverage follow separately. Source-PR: dotnet#9903 Source-Commit: e9dba90
Read one native storage page per catalog call, preserve canonical journal identity mapping, and scope continuations to each prefix and initialized provider. Publish Azure Blob catalog access after startup succeeds. Document traversal, memory and service-work bounds and the deferred Redis capability. Add offline conformance coverage for paging, tokens, cancellation, lifecycle, mappings, and preserved sorted listing across the built-in paged providers. Adapt the Azure paging proposal authored by Reuben Bond in e9dba90, and include the newer upstream S3 implementation. Source-PR: dotnet#9903 Source-Commit: e9dba90
Use the same configured-container and WAL identity traversal for ListAsync and ReadPageAsync. Delegate and factory object identity do not determine whether that traversal is supported, so remove the paging-only rejection which broke automatic paged discovery for equivalent custom configurations. Cover an equivalent WAL naming delegate and a wrapper container factory through the existing catalog object's paged capability, comparing paged jobs/shards identities with ListAsync across continuation pages. Keep token and API contracts unchanged.
Keep IJournalStorageCatalog as the single discovery abstraction. Replace the prefix parameter with extensible Prefix-only options and yield identities in provider traversal order while backend paging stays internal. Remove the separate public page interface, page results, authenticated continuation tokens, and volatile ordering index. Stream Azure/S3 pages and Redis metadata batches, retaining Redis duplicate suppression. Migrate callers and preserve explicit cancellation, lifecycle and identity semantics. BREAKING CHANGE: ListAsync now accepts JournalStorageCatalogOptions and returns provider traversal order. Callers requiring sorted results can materialize and sort explicitly. Scoped generated baseline suppressions cover the approved interface and volatile signature changes.
Check cancellation after Redis metadata-recovery script responses and before catalog completion, including a final concurrently deleted journal. Preserve native request behavior without retry or abort wrappers. Cover the simplified enumerable contract across built-in providers: first yields before later pages, early disposal, options snapshots, empty-page traversal, live existence, mappings, startup boundaries and Redis streaming/deduplication. Replace the retired public-page expectations with identity-level streaming assertions.
Scope the options name to the listing operation. Update provider signatures, callers, tests and documentation without changing enumeration behavior. Regenerate the public API and the existing scoped compatibility suppression targets.
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It introduces an intentional public API breaking change and rewires enumeration semantics across multiple storage providers, which warrants final human validation despite strong test coverage.
Review tier: Lite
Findings: None
What changed in this PR
This PR updates Orleans journaling catalog enumeration to stream results incrementally (instead of collecting + sorting the full catalog), reducing memory usage and improving time-to-first-result across built-in providers. It introduces an extensible ListOptions parameter (with Prefix) and changes the observable ordering from lexicographic to provider traversal order.
Changes:
- Replace
ListAsync(JournalId prefix, ...)withListAsync(ListOptions? options = null, ...)onIJournalStorageCatalogand update all in-repo callers/providers accordingly. - Rework Azure Blob, Azure Table, S3, Redis, and Volatile providers to yield identities as they are discovered (paging internally and avoiding whole-catalog buffering/sorting).
- Add/adjust conformance and provider-specific tests for ordering change, cancellation, early disposal, and option snapshot semantics; update docs and package compatibility suppressions/API surface.
| File | Description |
|---|---|
| test/Orleans.Journaling.Tests/VolatileJournalStorageProviderTests.cs | Updates tests to pass ListOptions and to assert results independent of traversal order. |
| test/Orleans.Journaling.Tests/S3JournalStorageTests.cs | Updates S3 tests to use ListOptions and to sort/assert without relying on provider ordering. |
| test/Orleans.Journaling.Tests/JournalStorageCatalogTests.cs | Adds offline catalog conformance tests for option snapshotting, paging, cancellation, early disposal, and error propagation across providers. |
| test/Orleans.Journaling.Tests/AzureTableJournalStorageProviderTests.cs | Updates Azure Table tests for new signature and provider traversal ordering expectations. |
| test/Orleans.DurableJobs.Tests/DurableJobs/JournaledJobShardManagerTests.cs | Updates Durable Jobs tests for the new ListAsync(ListOptions?) signature. |
| test/Extensions/Orleans.Redis.Tests/Journaling/RedisJournalStorageTests.cs | Updates Redis journaling tests to use ListOptions and relax ordering assertions. |
| test/Extensions/Orleans.Redis.Tests/Journaling/RedisJournalStorageCatalogTests.cs | Adds Redis-specific catalog tests for batching, prefix snapshotting, cancellation, disposal, and failure timing. |
| src/Redis/Orleans.Journaling.Redis/RedisJournalStorageProvider.cs | Streams SCAN results, reads metadata in bounded batches, suppresses duplicate journal ids, and yields incrementally. |
| src/Redis/Orleans.Journaling.Redis/README.md | Documents new discovery behavior, batching semantics, ordering, and cancellation/error behavior. |
| src/Orleans.Journaling/VolatileJournalStorage.cs | Updates volatile provider catalog enumeration to yield incrementally in traversal order (no sorting/buffering). |
| src/Orleans.Journaling/README.md | Documents the new catalog enumeration contract, ordering, and provider behaviors. |
| src/Orleans.Journaling/ListOptions.cs | Introduces the new options type (currently containing Prefix). |
| src/Orleans.Journaling/IJournalStorageCatalog.cs | Updates the public contract/signature and documents traversal-order + streaming semantics. |
| src/Orleans.Journaling/CompatibilitySuppressions.xml | Adds package validation suppressions for the intentional interface/signature breaking change. |
| src/Orleans.DurableJobs/JournaledJobShardManager.cs | Updates shard discovery to pass ListOptions for prefix filtering. |
| src/Azure/Orleans.Journaling.AzureStorage/README.md | Documents catalog enumeration behavior for Azure Blob/Table providers. |
| src/Azure/Orleans.Journaling.AzureStorage/AzureTableJournalStorageProvider.cs | Updates table provider enumeration to page and yield incrementally (no whole-list sort). |
| src/Azure/Orleans.Journaling.AzureStorage/AzureBlobJournalStorageProvider.cs | Updates blob provider enumeration to page and yield incrementally; publishes catalog access after successful initialization. |
| src/AWS/Orleans.Journaling.S3/S3JournalStorageProvider.cs | Updates S3 provider enumeration to yield identities per page without sorting/buffering; factors canonical WAL filtering into a helper. |
| src/AWS/Orleans.Journaling.S3/README.md | Documents traversal-order enumeration and paging/cancellation characteristics for S3. |
| src/api/Orleans.Journaling/Orleans.Journaling.cs | Updates generated API surface to reflect the new signature and ListOptions type. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Journal catalog implementations currently collect and sort the complete catalog before yielding identities. This increases discovery memory use and delays the first useful result until the storage traversal finishes.
Changes
Keep
IJournalStorageCatalog.ListAsyncas the single asynchronous enumeration API and replace its prefix parameter with an extensibleListOptionsobject containingPrefix. Built-in volatile, Azure Blob, Azure Table, S3, and Redis providers yield matching identities incrementally in provider traversal order, with storage pagination handled internally.Azure and S3 retain only their current native page during traversal. Volatile enumerates its existing storage dictionary and checks journal existence under the store lock. Redis reads metadata in batches of up to 128 keys and suppresses repeated matching identities with a seen-id set. Cancellation, early disposal, identity mappings, and configured Blob catalog behavior are covered by offline conformance regressions. Blob catalog access is published after initialization succeeds; Redis observes cancellation after its final metadata-recovery response and before completion.
Contract and migration
This intentionally changes the public signature to
ListAsync(ListOptions? options = null, CancellationToken cancellationToken = default)and changes ordering from lexicographic journal-id order to provider traversal order. Existing prefix callers passnew ListOptions { Prefix = prefix }; applications requiring ordinal order can materialize and sort explicitly. Generated API changes and package-scoped compatibility suppressions accompany the approved signature break.One enumerator advance can cross several empty or filtered storage pages. Consumer batching therefore bounds yielded identities and metadata operations; backend requests, scan work, and latency remain service-defined. Redis retains O(N) seen-id memory and its SCAN count is a hint. Enumeration observes live storage, and a listing failure is surfaced to the caller before a later traversal is started. Expected benefits are earlier results and lower traversal memory for Azure/S3; total sweep I/O and throughput improvements are workload-dependent.
Scope and attribution
This is the library-focused catalog extraction from #9903, with mechanical caller updates. The source paging proposal originated in
e9dba90d8b87b15254b29212e7f5de344eab820e, authored by Reuben Bond during review of KSemenenko's advanced reminder proposal. Review simplified it to the existing enumerable API; current upstream S3 support is included. Incremental Durable Jobs discovery is a dependent change.External-service integration and performance improvements remain unmeasured.
Microsoft Reviewers: Open in CodeFlow