feat(membership): adapt probe timeouts

[ReubenBond]

Problem

Membership probing uses the same fixed timeout and cadence for every peer despite differing network conditions. This can delay failure detection for responsive peers while giving slower peers too little tolerance.

Solution

• retain ProbeTimeout as the initial per-peer timeout
• add configurable minimum and maximum bounds, defaulting to half and four times ProbeTimeout
• learn a timeout independently for each peer from successful direct-probe RTTs using a bounded Phi Accrual model
• adapt probe cadence to the latest effective timeout, measuring the delay from the previous probe attempt start so quick probes wait the remainder and timed-out probes retry immediately
• preserve indirect-probe, local-health, debugger, connection-liveness, and conservative failure-budget behavior
• document the protocol and validate option/API compatibility

Rationale

Peer-local evidence allows responsive links to fail faster while giving consistently slower links more tolerance. Failed and indirect probes are excluded because they are censored measurements or represent a different network path. Tying cadence to the learned timeout lets shorter estimates improve detection latency without allowing probes from a monitor to overlap.

[Copilot]

Pull request overview

This PR updates Orleans membership failure detection to decouple probe cadence from probe response deadlines and to adapt direct-probe timeouts per peer using a bounded Phi Accrual model. It updates runtime logic, configuration validation, public API surface, documentation, and related tests to reflect the new probe timeout semantics.

Changes:

• Split ProbeTimeout into ProbeInterval + InitialProbeTimeout, with min/max bounds and an obsolete compatibility alias.
• Add a per-peer adaptive timeout model (Phi Accrual) used for successful direct probes, with clamping and existing extensions (local health, indirect hop, debugger).
• Update tests, failure-detection time calculations, and documentation to align with the new behavior.

Show a summary per file

File	Description
test/Orleans.Runtime.Internal.Tests/MembershipTests/ClientIdPartitionDataRebuildTests.cs	Updates test configuration to use the new probe interval/timeout options.
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryLeaseTests.cs	Adjusts lease-duration expectations to reflect worst-case failure detection time.
test/Orleans.Core.Tests/SiloBuilderTests.cs	Adds configuration validation tests for new probe interval/timeout constraints.
test/Orleans.Core.Tests/Membership/SiloHealthMonitorTests.cs	Extends coverage for adaptive direct-probe timeout behavior and ensures indirect/failed probes don’t feed adaptation.
test/Orleans.Core.Tests/Membership/PhiAccrualFailureDetectorTests.cs	Adds unit tests for the Phi Accrual failure detector behavior and clamping/extension ordering.
test/Orleans.Core.Tests/Membership/MembershipAgentTests.cs	Updates connectivity validation test setup to use new probe interval/initial timeout.
test/Orleans.Core.Tests/Membership/ClusterMembershipOptionsTests.cs	Adds tests for defaults, legacy alias behavior, and configuration precedence for new options.
src/Orleans.TestingHost/TestCluster.cs	Updates stabilization-time calculation to use the maximum probe cycle time.
src/Orleans.TestingHost/InProcTestCluster.cs	Same stabilization-time update for in-proc test clusters.
src/Orleans.Runtime/MembershipService/SiloMetadata/SiloMetadaCache.cs	Uses failure-detection timeout helper for negative cache period calculation.
src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs	Implements per-peer adaptive timeout calculation and switches timer cadence to ProbeInterval.
src/Orleans.Runtime/MembershipService/ProbingSiloHealthMonitor.cs	Uses failure-detection timeout helper for probe request/response recency windows.
src/Orleans.Runtime/MembershipService/PhiAccrualFailureDetector.cs	Introduces Phi Accrual detector implementation and bounded history for RTT samples.
src/Orleans.Runtime/MembershipService/MembershipAgent.cs	Uses ProbeInterval for retry cadence and InitialProbeTimeout for probe deadline.
src/Orleans.Runtime/MembershipService/LocalSiloHealthMonitor.cs	Uses failure-detection timeout helper for recency window logic.
src/Orleans.Runtime/MembershipService/ClusterHealthMonitor.cs	Uses MaxProbeTimeout for shutdown cancellation and failure-detection timeout for liveness monitoring window.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs	Uses failure-detection timeout helper for dead-silo lease duration calculation.
src/Orleans.Runtime/Configuration/Validators/SiloClusteringValidator.cs	Adds validation for probe interval/timeout bounds and max-cycle-time overflow safety.
src/Orleans.Runtime/Configuration/ClusterMembershipOptionsExtensions.cs	Adds helpers to compute maximum probe cycle time and overall failure detection timeout.
src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs	Adds new probe interval/initial/min/max options and marks ProbeTimeout obsolete as a compatibility alias.
src/api/Orleans.Core/Orleans.Core.cs	Updates generated public API surface for new ClusterMembershipOptions properties.
docs/site/src/content/docs/implementation/cluster-management.md	Documents the updated protocol and adaptive timeout behavior and updates option defaults.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 22/22 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs:81

• MinProbeTimeout defaults to TimeSpan.FromTicks(InitialProbeTimeout.Ticks / 2), which becomes TimeSpan.Zero when InitialProbeTimeout is set to 1 tick. That makes the options invalid under SiloClusteringValidator (it requires MinProbeTimeout > 0) even though InitialProbeTimeout > 0 is allowed. Consider rounding up/clamping to at least 1 tick so the derived default stays valid for all positive initial timeouts.

        public TimeSpan MinProbeTimeout
        {
            get => _minProbeTimeout ?? TimeSpan.FromTicks(InitialProbeTimeout.Ticks / 2);
            set => _minProbeTimeout = value;
        }

• Files reviewed: 31/31 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 32/32 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 34/34 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:291

• CalculateIndirectProbeTargetTimeout can return TimeSpan.Zero for very small timeouts (e.g., 1 tick) because it always subtracts at least 1 tick of response allowance. Passing a zero/negative direct-probe timeout to the intermediary can cause immediate failures or downstream ArgumentOutOfRangeException.

            var extensionFactor = 1 + localDegradationScore;
            var responseAllowanceTicks = Math.Max(1, timeout.Ticks / (extensionFactor + 1));
            return TimeSpan.FromTicks(timeout.Ticks - responseAllowanceTicks);

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:279

• When the learned timeout falls below MinProbeTimeout, the effective timeout is clamped up to MinProbeTimeout, but the debugger extension is still computed from the unclamped learned timeout. This can under-extend timeouts under a debugger (relative to the effective/clamped baseline) when MinProbeTimeout > learned timeout, increasing the chance of false failure detection during debugging.

            var timeout = failureDetector.GetTimeout(options.MinProbeTimeout, options.MaxProbeTimeout, extensionFactor);
            if (isDebuggerAttached)
            {
                var debuggerExtensionTicks = failureDetector.GetTimeout().Ticks * 25d;
                var extendedTimeoutTicks = Math.Min(MaxSupportedTimerTimeout.Ticks, timeout.Ticks + debuggerExtensionTicks);
                timeout = TimeSpan.FromTicks((long)extendedTimeoutTicks);

• Files reviewed: 22/22 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs:47

• MinProbeTimeout default is derived using integer tick division (ProbeTimeout.Ticks / 2), which can yield TimeSpan.Zero for very small ProbeTimeout values (eg TimeSpan.FromTicks(1)), making the default bounds invalid even though ProbeTimeout > 0. Consider clamping the computed default to at least 1 tick to keep the derived minimum valid for all positive probe timeouts.

        public TimeSpan MinProbeTimeout
        {
            get => _minProbeTimeout ?? TimeSpan.FromTicks(ProbeTimeout.Ticks / 2);
            set => _minProbeTimeout = value;
        }

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:294

• Debugger timeout extension is computed from failureDetector.GetTimeout() (unclamped/unextended) rather than from the effective clamped base. If the learned timeout falls below MinProbeTimeout, this can significantly under-extend the timeout while debugging (eg min clamp to 5s but learned 100ms -> only +2.5s). Consider basing the debugger extension on the clamped baseline timeout (eg extensionFactor: 1) so it remains consistent with configured bounds.

            var timeout = failureDetector.GetTimeout(options.MinProbeTimeout, options.MaxProbeTimeout, extensionFactor);
            if (isDebuggerAttached)
            {
                var debuggerExtensionTicks = failureDetector.GetTimeout().Ticks * 25d;
                var extendedTimeoutTicks = Math.Min(MaxSupportedTimerTimeout.Ticks, timeout.Ticks + debuggerExtensionTicks);

• Files reviewed: 20/20 changed files
• Comments generated: 0 new
• Review effort level: Lite