feat(manifest): enable content-addressed retrieval by default

docs/site/src/content/docs/host/monitoring/metrics-catalog.md

@@ -86,6 +86,18 @@ Request latency covers the interval until the caller's callback completes, inclu
 | `orleans-scheduler-long-running-turns` | C | Turns, implicit | - | Grain micro-turns whose synchronous execution exceeded <xref:Orleans.Configuration.SchedulingOptions.TurnWarningLengthThreshold>. |
 | `orleans-system-targets` | UDC | System targets, implicit | `type` | Current Orleans system-target instances by type. |
 
+## Cluster manifests
+
+| Instrument | Type | Unit | Attributes | Description |
+|---|---|---|---|---|
+| `orleans-manifest-cache-lookups` | C | Lookups, implicit | `result`, `source` | Content-cache lookups with `hit` or `miss` results, sourced from a `silo` hash response or a `peer` summary. |
+| `orleans-manifest-fallbacks` | C | Retrievals, implicit | `reason` | Hash retrievals falling back to the direct manifest RPC: `error`, `missing`, or `mismatch`. |
+| `orleans-manifest-peer-probes` | C | Attempts, implicit | `status` | Local peer attempts completed with `success`, `timeout`, `canceled`, or `error`; `skipped` counts attempts denied by local admission. |
+| `orleans-manifest-peer-repairs` | C | Silo entries, implicit | - | Missing silo manifest entries supplied by successfully published peer repairs. |
+| `orleans-manifest-retrieval-duration` | H | `ms` | `mode`, `status` | Local silo-manifest retrieval duration in `direct` or `hash` mode, ending in `success`, `error`, or `canceled`. |
+
+The categories are fixed and carry no silo addresses, manifest hashes, or grain type names. Probe cancellation includes optional work superseded by complete direct results. See [cluster manifest retrieval](../../implementation/cluster-manifest-retrieval.md) for publication, fallback, and rollout semantics.
+
 ## Grain directory and consistent rings
 
 | Instrument | Type | Unit | Attributes | Description |

docs/site/src/content/docs/implementation/cluster-manifest-retrieval.md

@@ -0,0 +1,62 @@
+---
+title: Cluster manifest retrieval
+description: How Orleans discovers silo manifests, reuses content hashes, and repairs missing metadata during membership changes.
+ms.date: 09/10/2026
+ms.topic: concept-article
+---
+
+# Cluster manifest retrieval
+
+Each silo publishes an immutable grain manifest describing its grain types, interfaces, and properties. The cluster manifest provider assembles these manifests into the metadata used for type resolution and version-aware placement.
+
+## Membership and publication
+
+Cluster manifests have a major version corresponding to the membership version and a minor version which advances as manifests arrive. When membership advances, the provider synchronously prunes non-active silos and includes the local silo when it is active. Local grain metadata also remains available while the local silo is starting.
+
+The provider retrieves missing active silos' manifests and publishes successful results. Unsuccessful fetches are retried after five seconds or when a newer membership snapshot arrives. Cancellation reaches the remote call and stops local waiting.
+
+## Content-addressed retrieval
+
+<xref:Orleans.Configuration.ClusterManifestOptions.EnableContentAddressedRetrieval> defaults to `true`, enabling hash-based retrieval and peer repair. Set it to `false` in <xref:Orleans.Configuration.ClusterManifestOptions> to request each missing active silo's manifest directly. Configure the option through the silo builder before starting the silo. The provider captures the value at construction; a restart applies configuration changes.
+
+With this option enabled, the provider asks each missing silo for the content hash of its local manifest. A matching entry in the local hash cache supplies the manifest immediately. Otherwise, the provider requests the manifest by hash, verifies its content, and adds it to the cache.
+
+Hashes use incremental SHA-256 over a fixed canonical traversal: an encoding version, sorted grain entries, then sorted interface entries. Each section and property collection starts with its count. Identifiers have a byte-length prefix; property keys and values have UTF-16 code-unit-length prefixes. A length of `-1` represents null or a default identifier, while `0` represents an empty value. Counts and lengths are signed 32-bit big-endian integers, and UTF-16 code units are written in big-endian order. This layout preserves exact identifier bytes and string contents, including invalid UTF-16, and gives equivalent content the same hash regardless of dictionary insertion order. Hashes of immutable manifest instances are memoized with weak keys, allowing the manifests to be collected when their owners release them.
+
+Each publication creates a fresh cache containing its live manifests and local metadata. In-flight retrievals retain their original cache instance. A stale retrieval can populate that instance, while newer publications retain their own cache. The provider exposes the new manifest version before its new cache. Retrievals capture the cache before checking the version, so an older update which captures the newer cache also observes that its membership snapshot has been superseded.
+
+## Peer repair and bounded waiting
+
+When more than one active silo's manifest is missing, the enabled provider also probes up to three peers selected from a rotating ordered membership list. Direct retrieval starts alongside these probes.
+
+Each probe has a one-second deadline shared by its hash-summary and manifest-update requests. Cached hashes can satisfy missing entries directly; otherwise the provider requests a complete update from the peer and verifies each candidate against the summary's hash. Successfully repaired entries are published immediately, including partial repairs. Direct requests for those entries are removed from the required completion set, allowing the remaining successful fetches to advance the manifest independently.
+
+When direct retrieval supplies every missing manifest first, the provider cancels the optional peer attempts and publishes the direct results immediately. The successful direct path therefore completes independently of a slow peer-summary request.
+
+The provider admits at most three concurrent local probe attempts. Completion, timeout, and caller cancellation release the attempt's slot immediately, allowing later retries to proceed. The one-second deadline cancels the request token and ends local waiting; Orleans signals cancellation to the peer using the ordinary RPC path. Direct retrieval continues alongside these attempts. Late responses leave the completed attempt's result unchanged, and late failures are observed and logged. Retry selection rotates through active peers, with retries paced by the five-second delay or a newer membership snapshot.
+
+## Compatibility, rollout, and rollback
+
+Silos serve hash requests on demand, including when they are configured for direct retrieval. Providers using the default content-addressed mode compute and reuse content hashes as manifests become available.
+
+During a rolling deployment, enabled silos try hash retrieval and fall back to the established direct RPC when a peer rejects the newer method or its hash request fails. An invalid or unavailable hash-addressed body also triggers direct retrieval. Independent remote cancellation follows that compatibility path; cancellation of the local request propagates to the caller. Peer-repair failures leave direct retrieval responsible for filling the missing entries.
+
+Upgraded silos adopt content-addressed retrieval at startup while existing silos continue using their deployed implementation. Unsupported hash requests can produce transient exception logs and extra requests during the upgrade; successful direct fallback supplies the required metadata. Individual silo-manifest RPCs retain the configured system response timeout, while optional peer-summary/update attempts have a one-second deadline.
+
+Observe manifest retrieval and peer-probe diagnostics during joins and restarts. Debug logs distinguish hash fallback, peer timeout, occupied probe slots, and late failures. Warnings identify failed direct fetches. To select direct retrieval, set the option to `false` and restart the affected silos; they continue answering enabled peers' hash requests on demand.
+
+## Measuring retrieval
+
+The `Microsoft.Orleans` meter exposes [manifest retrieval instruments](../host/monitoring/metrics-catalog.md#cluster-manifests). Compare a canary with silos using direct retrieval during equivalent joins and restarts:
+
+| Signal | Interpretation |
+|---|---|
+| `orleans-manifest-cache-lookups`, split by `result` and `source` | The hit fraction shows how often known content satisfies direct hash requests or peer-summary lookups. |
+| `orleans-manifest-fallbacks`, split by `reason` | Counts hash retrievals which proceed through the direct RPC after an error, missing body, or content mismatch. |
+| `orleans-manifest-peer-probes`, split by `status` | Shows completed local attempts, timeouts, cancellation, errors, and admission skips. Cancellation also includes optional probes superseded by successful direct retrieval. |
+| `orleans-manifest-peer-repairs` | Counts missing silo entries supplied by successfully published peer repairs. Repeated summaries contribute each repaired entry once per publication. |
+| `orleans-manifest-retrieval-duration`, split by `mode` and `status` | Measures each local silo-manifest retrieval in milliseconds, including cache lookup, fallback, and terminal cancellation or failure. |
+
+Attributes use fixed categories, keeping time-series cardinality stable across silo restarts. The direct mode records retrieval duration while the hash and peer counters reflect content-addressed retrieval activity. Use these metrics alongside serialized message sizes and silo CPU/memory to evaluate the benefit for the service's manifest sizes and mix of application versions.
+
+See [cluster membership](cluster-management.md) for membership transitions and [rolling version skew](rolling-version-skew.md) for how manifest metadata drives interface-version selection.

docs/site/src/content/docs/toc.yml

@@ -343,6 +343,8 @@ items:
         href: implementation/grain-directory.md
       - name: Cluster membership
         href: implementation/cluster-management.md
+      - name: Cluster manifest retrieval
+        href: implementation/cluster-manifest-retrieval.md
       - name: Runtime lifecycle
         href: implementation/orleans-lifecycle.md
       - name: Scheduling and turn execution

src/Orleans.Core/Diagnostics/Metrics/ClusterManifestInstruments.cs

@@ -0,0 +1,38 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics.Metrics;
+
+namespace Orleans.Runtime;
+
+internal sealed class ClusterManifestInstruments(OrleansInstruments instruments)
+{
+    private readonly Counter<long> _cacheLookups = instruments.Meter.CreateCounter<long>(InstrumentNames.MANIFEST_CACHE_LOOKUPS);
+    private readonly Counter<long> _fallbacks = instruments.Meter.CreateCounter<long>(InstrumentNames.MANIFEST_FALLBACKS);
+    private readonly Counter<long> _peerProbes = instruments.Meter.CreateCounter<long>(InstrumentNames.MANIFEST_PEER_PROBES);
+    private readonly Counter<long> _peerRepairs = instruments.Meter.CreateCounter<long>(InstrumentNames.MANIFEST_PEER_REPAIRS);
+    private readonly Histogram<double> _retrievalDuration = instruments.Meter.CreateHistogram<double>(InstrumentNames.MANIFEST_RETRIEVAL_DURATION, "ms");
+
+    public bool RetrievalDurationEnabled => _retrievalDuration.Enabled;
+
+    public void OnCacheLookup(bool hit, string source) => _cacheLookups.Add(
+        1,
+        new KeyValuePair<string, object?>("result", hit ? "hit" : "miss"),
+        new KeyValuePair<string, object?>("source", source));
+
+    public void OnFallback(string reason) => _fallbacks.Add(1, new KeyValuePair<string, object?>("reason", reason));
+
+    public void OnPeerProbe(string status) => _peerProbes.Add(1, new KeyValuePair<string, object?>("status", status));
+
+    public void OnPeerRepair(int count) => _peerRepairs.Add(count);
+
+    public void OnRetrievalCompleted(TimeSpan elapsed, string mode, string status)
+    {
+        if (_retrievalDuration.Enabled)
+        {
+            _retrievalDuration.Record(
+                elapsed.TotalMilliseconds,
+                new KeyValuePair<string, object?>("mode", mode),
+                new KeyValuePair<string, object?>("status", status));
+        }
+    }
+}

src/Orleans.Core/Diagnostics/Metrics/InstrumentNames.cs

@@ -38,6 +38,13 @@ internal static class InstrumentNames
     // Runtime
     public const string SCHEDULER_NUM_LONG_RUNNING_TURNS = "orleans-scheduler-long-running-turns";
 
+    // Cluster manifests
+    public const string MANIFEST_CACHE_LOOKUPS = "orleans-manifest-cache-lookups";
+    public const string MANIFEST_FALLBACKS = "orleans-manifest-fallbacks";
+    public const string MANIFEST_PEER_PROBES = "orleans-manifest-peer-probes";
+    public const string MANIFEST_PEER_REPAIRS = "orleans-manifest-peer-repairs";
+    public const string MANIFEST_RETRIEVAL_DURATION = "orleans-manifest-retrieval-duration";
+
     // Catalog
     public const string CATALOG_ACTIVATION_COUNT = "orleans-catalog-activations";
     public const string CATALOG_ACTIVATION_WORKING_SET = "orleans-catalog-activation-working-set";

src/Orleans.Core/Manifest/IClusterManifestSystemTarget.cs

@@ -1,3 +1,4 @@
+using System.Collections.Generic;
 using System.Collections.Immutable;
 using System.Threading;
 using System.Threading.Tasks;
@@ -13,16 +14,98 @@ internal interface IClusterManifestSystemTarget : ISystemTarget
         /// <summary>
         /// Gets the current cluster manifest.
         /// </summary>
+        /// <param name="cancellationToken">The token used to cancel the request.</param>
         /// <returns>The current cluster manifest.</returns>
         [Alias("40D39F85")]
         ValueTask<ClusterManifest> GetClusterManifest(CancellationToken cancellationToken = default);
 
         /// <summary>
         /// Gets an updated cluster manifest if newer than the provided <paramref name="previousVersion"/>.
         /// </summary>
+        /// <param name="previousVersion">The last observed manifest version.</param>
+        /// <param name="cancellationToken">The token used to cancel the request.</param>
         /// <returns>The current cluster manifest, or <see langword="null"/> if it is not newer than the provided version.</returns>
         [Alias("4EFCA109")]
         ValueTask<ClusterManifestUpdate?> GetClusterManifestUpdate(MajorMinorVersion previousVersion, CancellationToken cancellationToken = default);
+
+        /// <summary>
+        /// Gets a hash summary for the current cluster manifest.
+        /// </summary>
+        /// <param name="cancellationToken">The token used to cancel the request.</param>
+        /// <returns>The current cluster manifest hash summary.</returns>
+        [Alias("25AE6E4A")]
+        ValueTask<ClusterManifestHashSummary> GetClusterManifestHashSummary(CancellationToken cancellationToken);
+
+        /// <summary>
+        /// Gets the hash of the local silo manifest.
+        /// </summary>
+        /// <param name="cancellationToken">The token used to cancel the request.</param>
+        /// <returns>The hash of the local silo manifest.</returns>
+        [Alias("3D9B7FE6")]
+        ValueTask<ManifestHash> GetSiloManifestHash(CancellationToken cancellationToken);
+
+        /// <summary>
+        /// Gets the local silo manifest if the provided hash matches it.
+        /// </summary>
+        /// <param name="hash">The expected manifest hash.</param>
+        /// <param name="cancellationToken">The token used to cancel the request.</param>
+        /// <returns>The local silo manifest, or <see langword="null"/> if the hash does not match.</returns>
+        [Alias("93B8854F")]
+        ValueTask<GrainManifest?> GetSiloManifestByHash(ManifestHash hash, CancellationToken cancellationToken);
+    }
+
+    /// <summary>
+    /// Identifies a manifest by its canonical content hash.
+    /// </summary>
+    [GenerateSerializer, Immutable]
+    internal readonly struct ManifestHash : System.IEquatable<ManifestHash>
+    {
+        public ManifestHash(string value)
+        {
+            Value = value;
+        }
+
+        [Id(0)]
+        public string Value { get; }
+
+        public bool Equals(ManifestHash other) => string.Equals(Value, other.Value, System.StringComparison.Ordinal);
+
+        public override bool Equals(object? obj) => obj is ManifestHash other && Equals(other);
+
+        public override int GetHashCode() => System.StringComparer.Ordinal.GetHashCode(Value ?? string.Empty);
+
+        public override string ToString() => Value ?? string.Empty;
+
+        public static bool operator ==(ManifestHash left, ManifestHash right) => left.Equals(right);
+
+        public static bool operator !=(ManifestHash left, ManifestHash right) => !left.Equals(right);
+    }
+
+    /// <summary>
+    /// Represents a hash summary for a cluster manifest.
+    /// </summary>
+    [GenerateSerializer, Immutable]
+    internal sealed class ClusterManifestHashSummary
+    {
+        public ClusterManifestHashSummary(
+            MajorMinorVersion version,
+            Dictionary<SiloAddress, ManifestHash> siloManifestHashes)
+        {
+            Version = version;
+            SiloManifestHashes = siloManifestHashes.ToImmutableDictionary();
+        }
+
+        /// <summary>
+        /// Gets the cluster manifest version.
+        /// </summary>
+        [Id(0)]
+        public MajorMinorVersion Version { get; }
+
+        /// <summary>
+        /// Gets the manifest hash for each silo.
+        /// </summary>
+        [Id(1)]
+        public ImmutableDictionary<SiloAddress, ManifestHash> SiloManifestHashes { get; }
     }
 
     /// <summary>

src/Orleans.Core/OrleansContracts.txt

@@ -49,7 +49,10 @@ interface [GrainInterfaceType("Orleans.Placement.Repartitioning.IActivationRepar
 
 interface [GrainInterfaceType("Orleans.Runtime.IClusterManifestSystemTarget")] Orleans.Runtime.IClusterManifestSystemTarget [Version(0)]
   40D39F85: GetClusterManifest(System.Threading.CancellationToken) -> ValueTask<Orleans.Metadata.ClusterManifest>
+  25AE6E4A: GetClusterManifestHashSummary(System.Threading.CancellationToken) -> ValueTask<Orleans.Runtime.ClusterManifestHashSummary>
   4EFCA109: GetClusterManifestUpdate(Orleans.Metadata.MajorMinorVersion, System.Threading.CancellationToken) -> ValueTask<Orleans.Runtime.ClusterManifestUpdate?>
+  93B8854F: GetSiloManifestByHash(Orleans.Runtime.ManifestHash, System.Threading.CancellationToken) -> ValueTask<Orleans.Metadata.GrainManifest?>
+  3D9B7FE6: GetSiloManifestHash(System.Threading.CancellationToken) -> ValueTask<Orleans.Runtime.ManifestHash>
 
 interface [GrainInterfaceType("Orleans.Runtime.IDeploymentLoadPublisher")] Orleans.Runtime.IDeploymentLoadPublisher [Version(0)]
   C5255F0C: UpdateRuntimeStatistics(Orleans.Runtime.SiloAddress, Orleans.Runtime.SiloRuntimeStatistics, System.Threading.CancellationToken) -> Task

src/Orleans.Runtime/Configuration/Options/ClusterManifestOptions.cs

@@ -0,0 +1,22 @@
+namespace Orleans.Configuration;
+
+/// <summary>
+/// Configures how a silo retrieves grain manifests from the cluster.
+/// </summary>
+public sealed class ClusterManifestOptions
+{
+    /// <summary>
+    /// Gets or sets a value indicating whether the silo retrieves manifests by content hash
+    /// and uses peer summaries to repair missing manifests.
+    /// </summary>
+    /// <value>
+    /// <see langword="true"/> by default. Set to <see langword="false"/> to retrieve each active silo's manifest directly.
+    /// </value>
+    /// <remarks>
+    /// The runtime captures this setting when the silo's manifest provider is constructed.
+    /// Restart the silo to apply a changed value. Silos serve hash requests from enabled peers
+    /// on demand, including when their own retrieval is configured to use the direct path.
+    /// Peer repair uses up to three concurrent local attempts, each with a one-second deadline.
+    /// </remarks>
+    public bool EnableContentAddressedRetrieval { get; set; } = true;
+}