fix(runtime): close callback admission during shutdown - #10290
Merged
ReubenBond merged 5 commits intoJul 21, 2026
Merged
Conversation
Hard-killed silos can cancel callback timer shutdown before outstanding callbacks are faulted, leaving stateless worker disposal blocked forever. Complete callbacks before the cancellation-sensitive wait and exercise the path with multiple in-flight workers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 10faef46-7323-49a5-bd75-9235f99dcf8f
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
ReubenBond
changed the title
Faulting outstanding transaction callbacks can resume code which issues follow-up calls after the shutdown sweep. Close callback registration when runtime shutdown begins so those calls fail immediately instead of blocking host disposal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 10faef46-7323-49a5-bd75-9235f99dcf8f
ReubenBond
changed the title
Close callback admission before draining outstanding calls in silo and client runtimes. Use pre- and post-insertion checks so every callback is completed by either the shutdown sweep or the admitting thread, and safely publish cancellation registrations across concurrent completion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 10faef46-7323-49a5-bd75-9235f99dcf8f
ReubenBond
changed the title
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 10faef46-7323-49a5-bd75-9235f99dcf8f
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 10faef46-7323-49a5-bd75-9235f99dcf8f
This was referenced Jul 22, 2026
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
Merged
This was referenced Aug 5, 2026
This was referenced Aug 18, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hard-killing an in-process silo can deadlock host disposal when stateless workers have transaction calls in flight.
The CI dumps show host disposal waiting on
StatelessWorkerGrainContext.DisposeAsyncInternal. Faulting the original callbacks resumes transaction cleanup, which can issue follow-up requests after the shutdown sweep and register callbacks that can never receive responses.Close callback admission before draining outstanding calls in both silo and client runtimes. Requests check the stopping state before insertion and again afterward, ensuring each callback is completed by either the shutdown sweep or the admitting thread. Callback cancellation registration now uses a nonblocking publication/disposal handshake so synchronous cancellation or concurrent completion cannot leak a registration.
The regression coverage includes multiple blocked silo workers which retry after shutdown, a client call which retries after shutdown, cancellation during registration, and registration attempted after callback completion.