Repository navigation
feat(manifest): enable content-addressed retrieval by default - #11239
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new public ClusterManifestOptions API is not reflected in the generated src/api surfaces, which is required for public API changes in packable projects.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions |
What changed in this PR
This PR introduces an opt-in, content-addressed cluster manifest retrieval path in Orleans.Runtime to avoid redundant full-manifest RPCs when immutable manifest content is already available locally, while preserving the legacy direct-fetch behavior by default for compatibility and rollout safety.
Changes:
- Add SHA-256 canonical manifest hashing and memoization, plus new manifest-hash RPC contracts to support hash-first retrieval and peer-assisted repair.
- Extend
ClusterManifestProviderwith publication-scoped manifest hash caches, bounded peer probing (3 concurrent probes, 1s deadlines), and verified hash-based retrieval with legacy fallback. - Add focused tests, documentation, and a dedicated CI workflow for manifest retrieval coverage and API artifact generation.
| File | Description |
|---|---|
| test/Orleans.Runtime.Internal.Tests/Manifest/ManifestHashCalculatorTests.cs | Adds determinism and framing/edge-case coverage for canonical manifest hashing. |
| test/Orleans.Runtime.Internal.Tests/Manifest/ClusterManifestHashSummarySerializationTests.cs | Verifies wire contracts expose cancellation and new hash-summary/hash identity round-trips. |
| test/Orleans.Core.Tests/Manifest/ClusterManifestProviderTests.cs | Adds/extends provider regressions for cancellation, peer repair, bounded probes, and fallback behavior. |
| src/Orleans.Runtime/Manifest/ManifestHashCalculator.cs | Implements canonical SHA-256 hashing over a versioned token frame. |
| src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs | Implements opt-in content-addressed retrieval, cache epochs, peer probing/repair, and cancellation/timeout semantics. |
| src/Orleans.Runtime/Hosting/DefaultSiloServices.cs | Registers option formatting for ClusterManifestOptions. |
| src/Orleans.Runtime/GrainTypeManager/ClusterManifestSystemTarget.cs | Serves hash-summary and hash-addressed manifest RPCs with version-scoped summary caching. |
| src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs | Adds a public option to enable/disable content-addressed retrieval (default-off). |
| src/Orleans.Core/OrleansContracts.txt | Updates system target contract map with new methods/aliases. |
| src/Orleans.Core/Manifest/IClusterManifestSystemTarget.cs | Adds new hash-related RPC methods and supporting types (ManifestHash, ClusterManifestHashSummary). |
| docs/site/src/content/docs/toc.yml | Adds a new documentation entry for cluster manifest retrieval. |
| docs/site/src/content/docs/implementation/cluster-manifest-retrieval.md | Documents retrieval modes, caching epochs, bounded probing, cancellation, and rollout strategy. |
| .github/workflows/manifest-retrieval.yml | Adds a focused CI workflow for manifest retrieval tests and API artifact generation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
A new public API (ClusterManifestOptions) was added but the corresponding generated src/api surface update is missing, which is required for Orleans PRs affecting public API.
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Inject the keyed system-timer provider so one-second probe deadlines advance correctly.
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
Code coverage
Report-only conclusion: improved. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Two moderate findings require cache fixes, and the documentation needs correction.
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The CI test-filter defect must be fixed, and the timeout diagnostic should be corrected.
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The broad runtime, wire-contract, cancellation, and manifest-retrieval changes require final human review.
Review tier: Lite
Findings: 1
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The probe cancellation-lifetime issue in ClusterManifestProvider.cs must be fixed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs — Peer probes do not retain canceled RPC callbacks |
Pre-existing issues (1)
| Severity | Finding |
|---|---|
|
src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs — Generated API surface not updated for new public ClusterManifestOptions View comment |
fca9ea4 to
459af7f
Compare
fdac23f to
bda2d0e
Compare
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The default-on distributed retrieval path changes wire behavior, caching, cancellation, and publication concurrency, warranting final maintainer review despite extensive coverage.
Review tier: Balanced
Findings: None
Problem
Silos retrieve complete manifests from each active peer even when the same immutable metadata is already available locally. The manifest optimization in #10236 can be reviewed and rolled out independently from its lifecycle and dissemination changes.
Solution
Add
ClusterManifestOptions.EnableContentAddressedRetrieval, enabled by default and captured when the silo manifest provider is constructed. Silos reuse canonical SHA-256 content hashes, maintain publication-scoped caches, and repair missing metadata using bounded peer probes. Explicitly setting the option tofalseselects the established direct silo-manifest RPC path. Servers answer hash requests on demand in either mode.During rolling upgrades, new silos attempt hash retrieval and fall back to the established direct RPC when an older peer rejects the new request or hash retrieval fails. Transient compatibility exceptions and additional requests are expected while versions coexist. Existing manifest RPC aliases remain stable, and explicit
falseconfiguration takes effect after restart. Individual manifest RPCs retain their configured system response timeout; optional peer probes use the one-second local deadline described below.Hash input follows a fixed canonical traversal: an encoding version, sorted grains, then sorted interfaces. Collection counts and nullable length prefixes delimit identifiers and property key/value pairs. Integers and exact UTF-16 code units use big-endian order;
-1and0distinguish null/default from empty. This keeps incremental hashing and weak identity memoization while removing the 21-token framing scheme. Fixed vectors and structural, ordering, raw-identifier, and string-boundary regressions cover the encoding.Peer repair admits up to three concurrent local attempts, each with a one-second deadline shared by its summary and update requests. Completion, timeout, and caller cancellation release the local attempt's slot immediately. Requests use ordinary cancellation tokens and signaling, and late failures are observed. Retry selection rotates through active peers and follows the existing five-second or membership-update retry cadence.
Complete direct results cancel optional peer attempts and publish immediately, so slow peer-summary requests do not delay successful direct retrieval. Partial repairs publish independently of redundant direct fetches, and stale fetches populate only their captured cache epoch.
Observability and maintainability
Expose five instruments on the existing
Microsoft.Orleansmeter: cache lookup outcomes, fallback reasons, local peer-attempt outcomes, published peer-repair yield, and retrieval duration. Attributes use bounded categories rather than silo addresses or content hashes. The architecture guide and metrics catalog explain how to evaluate these signals during rollout.Organize provider tests into cohesive partial files for baseline behavior, direct retrieval, peer repair, cancellation, timing, metrics, and shared setup. Centralized reflection helpers preserve the existing test structure and keep the production API unchanged. Cover default-enabled content reuse and startup capture, explicit opt-out, exact meter observations, simultaneous cold fetches, fast-direct/slow-peer timing, and real-provider join/departure and rolling default/opt-out mode changes. A real proxy verifies default-enabled fallback to a legacy-only system target; this exercises the old contract in the current runtime rather than a mixed-release binary deployment.
All manifest tests use the existing BVT/None CI partition. The generated Runtime API surface was produced by GenAPI and verified against its CI artifact; the temporary artifact workflow has been removed.
Independence
Extracted from #10236, independent of lifecycle #11237 and dissemination. Targets
mainwith no shared cancellation prerequisite. The implementation uses the established runtime cancellation behavior and keeps local probe admission within the manifest provider.Microsoft Reviewers: Open in CodeFlow