Skip to content

fix(runtime): support scoped cancellation acknowledgments - #11241

Closed
ReubenBond wants to merge 5 commits into
dotnet:mainfrom
ReubenBond:rb-fix-runtime-scoped-rpc-cancellation
Closed

ReubenBond wants to merge 5 commits into
dotnet:mainfrom
ReubenBond:rb-fix-runtime-scoped-rpc-cancellation

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

Problem

Runtime components which bound their own local waits need to retain the underlying caller RPC after signaling cancellation. With the default WaitForCancellationAcknowledgement = false, cancellation completes and unregisters that RPC immediately, allowing caller-side admission capacity to be released while the callee is still executing.

Solution

Add the internal Orleans.Runtime.CancellationAcknowledgementScope.Enter() opt-in around creation of a generated-proxy task. GrainReferenceRuntime snapshots and suppresses the scope before outgoing filters run, then explicitly carries the choice through the invocation pipeline into CallbackData. The callback combines this choice with the global acknowledgement policy.

Cancellation signals the callee while the opted-in RPC remains pending until a response, cancellation acknowledgement, response timeout, target failure, or local host shutdown. The configured response timeout and timeout-cancellation policy remain authoritative. Callers place their independently bounded local wait outside the invocation scope.

The policy lives in per-invocation runtime state, preserving generated request metadata, serialization aliases, and request reuse. Asynchronous outgoing filters retain the original invocation's choice, and additional RPCs initiated by those filters select their own policy. Scope disposal restores the prior execution-context value, including nested scopes and synchronous exceptions. Ordinary unscoped calls preserve the existing execution context without writing ambient state.

Rationale and scope

This is an independent shared runtime prerequisite for the split of #10236, alongside lifecycle prerequisite #11237 and manifest retrieval #11239. Feature call sites remain in their owning PRs. The isolatable runtime-and-regressions chain is dbdbcb348baa6cc0b21cdde4c8cd6fb47163f3cb, 2c7153dd7da980704640f8c01ae8efe08fe8ad80, and test-fixture correction 2133d59c381972c857a2392f2e32510699428bbb, based on main 025bf9ed87ae9636ca56ec9527e97b30e7171456.

Admission accounting tracks observable pending caller requests. Remote execution can continue after a runtime timeout; bounded physical remote execution additionally depends on transport delivery and callee cooperation.

Microsoft Reviewers: Open in CodeFlow

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The runtime changes span multiple invocation paths and require final human review.

Review tier: Lite
Findings: None

What changed in this PR

Adds opt-in, per-invocation cancellation acknowledgements while preserving existing global behavior.

Changes:

  • Propagates scoped cancellation policy through invocation and callback pipelines.
  • Adds scope, callback, filter, shutdown, and request-reuse tests.
  • Adds focused CI coverage.
File Summary
test/​Orleans.Runtime.Tests/​CancellationTests/​ScopedCancellationRpcTests.cs Tests scoped cancellation across generated proxies, filters, reuse, and shutdown.
test/​Orleans.Runtime.Tests/​CancellationTests/​CancellationAcknowledgementScopeTests.cs Tests scope nesting and execution-context restoration.
test/​Orleans.Runtime.Tests/​CallbackDataTests.cs Tests per-invocation callback acknowledgement behavior.
src/​Orleans.Runtime/​Core/​InsideRuntimeClient.cs Carries acknowledgement policy for silo calls.
src/​Orleans.Core/​Runtime/​OutsideRuntimeClient.cs Carries acknowledgement policy for client calls.
src/​Orleans.Core/​Runtime/​OutgoingCallInvoker.cs Preserves policy through outgoing filters.
src/​Orleans.Core/​Runtime/​IRuntimeClient.cs Extends the request dispatch contract.
src/​Orleans.Core/​Runtime/​GrainReferenceRuntime.cs Captures and propagates invocation policy.
src/​Orleans.Core/​Runtime/​CancellationAcknowledgementScope.cs Implements scoped policy management.
src/​Orleans.Core/​Runtime/​CallbackData.cs Applies per-callback cancellation policy.
.github/​workflows/​rpc-cancellation.yml Runs focused cancellation tests across frameworks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 10, 2026 21:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The runtime cancellation changes span multiple execution and callback paths and warrant final human review.

Review tier: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 10, 2026 22:04
@ReubenBond
ReubenBond marked this pull request as ready for review September 10, 2026 22:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Replace the execution-context identity assertions before approval.

Review tier: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 10, 2026 22:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Update the scope test to compare policy state rather than execution-context snapshot identity.

Review tier: Lite
Findings: None

@github-actions

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 81.51% (109,225 / 133,996) 81.54% (109,238 / 133,971) -0.0249 pp
Branches 70.40% (31,127 / 44,212) 70.35% (31,102 / 44,208) +0.0502 pp

Report-only conclusion: mixed.

The current-main baseline is commit cff49293e9 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

@ReubenBond

Copy link
Copy Markdown
Member Author

Closing per the user's direction. The consuming features will use ordinary cancellation and bounded local attempts, fan-out, message sizes, and retry pacing. Local concurrency limits describe active local attempts; they do not require tracking remote execution after a local timeout. The per-invocation acknowledgement policy and extra permit-retention machinery will be removed from the consuming PRs. This PR is retired without merging.

@ReubenBond ReubenBond closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants