Skip to content

Isolate background workflow dispatches from caller's cancellation token#7291

Open
avin3sh wants to merge 1 commit intoelsa-workflows:mainfrom
avin3sh:avin3sh-fix-bg-workflow-dispatcher
Open

Isolate background workflow dispatches from caller's cancellation token#7291
avin3sh wants to merge 1 commit intoelsa-workflows:mainfrom
avin3sh:avin3sh-fix-bg-workflow-dispatcher

Conversation

@avin3sh
Copy link
Contributor

@avin3sh avin3sh commented Feb 13, 2026

Child workflow dispatches fail when early-completing child workflows cancel the parent's cancellation token while dispatches are still in progress. This race condition is most visible with BulkDispatchWorkflows as it has potential to create a situation where with enough "items", some children complete while dispatches are still pending, but, technically, this bug affects all dispatch scenarios.

The solution here is to use CancellationToken.None for background commands - they should run independently:

await commandSender.SendAsync(command, CommandStrategy.Background, CreateHeaders(), CancellationToken.None);

This pattern already exists in the Bookmark Resume endpoint for the same reason,

// Some clients, like Blazor, may prematurely cancel their request upon navigation away from the page.
// In this case, we don't want to cancel the workflow execution.
// We need to better understand the conditions that cause this.
var workflowCancellationToken = CancellationToken.None;
await ResumeBookmarkedWorkflowAsync(payload, input, asynchronous, workflowCancellationToken);

Note: this fix complements #7218
This fix complements the separate CommandHandlerInvokerMiddleware fix:

When dispatching multiple child workflows, early child completions cancel the parent activity's token while dispatches are still in progress. This creates a race condition where remaining dispatches fail with OperationCanceledException.
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 13, 2026

Greptile Overview

Greptile Summary

Fixes race condition where child workflow dispatches fail when early-completing children cancel the parent's token while dispatches are still pending. Changed all four dispatch methods in BackgroundWorkflowDispatcher to use CancellationToken.None instead of the caller's token for background command execution, ensuring commands run independently of caller lifecycle. This follows the established pattern from the Bookmark Resume endpoint.

  • Replaced caller's cancellationToken with CancellationToken.None in all four commandSender.SendAsync() calls
  • Added explanatory comments: "Background commands run independently of caller's lifecycle"
  • Created comprehensive unit tests verifying token isolation for all dispatch methods
  • Complements PR Propagate exceptions from command handlers correctly #7218 which ensures clean OperationCanceledException handling in the command invoker middleware

The fix is particularly important for BulkDispatchWorkflows scenarios where multiple child workflows are dispatched concurrently, and some may complete before all dispatches finish.

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The change is well-focused, follows an established pattern from the Bookmark Resume endpoint, and includes comprehensive unit tests. The fix addresses a clear race condition with a simple, logical solution. The implementation is consistent across all four dispatch methods, and the tests verify the expected behavior.
  • No files require special attention

Important Files Changed

Filename Overview
src/modules/Elsa.Workflows.Runtime/Services/BackgroundWorkflowDispatcher.cs Changes all four dispatch methods to use CancellationToken.None instead of caller's token for background command execution, preventing race conditions when child workflows complete early
test/unit/Elsa.Workflows.Runtime.UnitTests/Services/BackgroundWorkflowDispatcherTests.cs New test file with four comprehensive tests verifying that each dispatch method correctly isolates background commands from caller's cancellation token

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Dispatcher as BackgroundWorkflowDispatcher
    participant NotificationSender
    participant CommandSender
    participant BackgroundQueue as Background Command Queue

    Caller->>Dispatcher: DispatchAsync(request, callerToken)
    Dispatcher->>NotificationSender: SendAsync(Dispatching notification, callerToken)
    NotificationSender-->>Dispatcher: 
    Dispatcher->>CommandSender: SendAsync(command, Background, headers, CancellationToken.None)
    Note over CommandSender,BackgroundQueue: Command queued with independent token
    CommandSender-->>Dispatcher: 
    Dispatcher->>NotificationSender: SendAsync(Dispatched notification, callerToken)
    NotificationSender-->>Dispatcher: 
    Dispatcher-->>Caller: Success response
    Note over Caller: Caller may cancel/complete here
    Note over BackgroundQueue: Background command continues<br/>execution independently
Loading

Last reviewed commit: 4204fc1

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile


await commandSender.SendAsync(command, CommandStrategy.Background, CreateHeaders(), cancellationToken);
// Background commands run independently of caller's lifecycle.
await commandSender.SendAsync(command, CommandStrategy.Background, CreateHeaders(), CancellationToken.None);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider applying this same pattern to BackgroundStimulusDispatcher and BackgroundWorkflowCancellationDispatcher for consistency, as they also dispatch background commands that should run independently of caller lifecycle.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prevents background workflow dispatches from being canceled by the caller’s CancellationToken, addressing a race where early-finishing child workflows (notably via BulkDispatchWorkflows) can cancel the parent token while dispatches are still being enqueued.

Changes:

  • Use CancellationToken.None when sending workflow dispatch commands with CommandStrategy.Background.
  • Add unit tests verifying caller tokens are not propagated to background command dispatch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/modules/Elsa.Workflows.Runtime/Services/BackgroundWorkflowDispatcher.cs Ensures background dispatch commands are enqueued independently from the caller’s cancellation token.
test/unit/Elsa.Workflows.Runtime.UnitTests/Services/BackgroundWorkflowDispatcherTests.cs Adds coverage to assert CancellationToken.None is used for background command sends across dispatcher overloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants