fix(core): Fix race condition when stopping jobs in queue mode

[ivov]

Summary

Fix the race condition on stopping an execution in queue mode where the main process calls job.moveToFailed() (which deletes the job key from Redis) while the worker's Bull processor is still running, causing Missing key for job updateProgress errors.

To reproduce:

1. Start main and worker in queue mode with OFFLOAD_MANUAL_EXECUTIONS_TO_WORKERS=true
2. Create a workflow: Webhook → Code node with await new Promise(r => setTimeout(r, 120_000))
3. Publish the workflow and trigger the webhook
4. In the UI, go to Executions → find the running execution → click Stop
5. Before fix: Worker logs Missing key for job <id> updateProgress and Missing key for job <id> failed, execution appears stuck in UI
6. After fix: Worker throws ManualExecutionCancelledError, Bull moves the job to failed cleanly, no missing key errors

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/CAT-2616

Review / Merge checklist

• PR title and summary are descriptive. (conventions)
• Docs updated or follow-up ticket created.
• Tests included.
• PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
packages/cli/src/scaling/job-processor.ts	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/cli/src/scaling/scaling.service.ts">

<violation number="1" location="packages/cli/src/scaling/scaling.service.ts:260">
P0: Custom agent: **Quality & Performance Review**

According to linked Linear issue CAT-2616, active-job cancellation still depends solely on Bull `global:progress` delivery. This change returns success immediately after publishing `abort-job`, so the documented "stop signal never reaches worker" failure mode remains and stopped executions can continue running.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant UI as User Interface
    participant Main as Main Process (ScalingService)
    participant Redis as Redis (Bull Queue)
    participant Worker as Worker Process (JobProcessor)

    Note over UI,Worker: Manual Stop Flow (Queue Mode)

    UI->>Main: Request Stop Execution
    Main->>Redis: job.isActive()
    Redis-->>Main: true

    rect rgb(23, 37, 84)
    Note over Main,Redis: CHANGED: Stop logic shifted to Worker
    Main->>Redis: job.progress({ kind: 'abort-job' })
    Note right of Main: No longer calls job.moveToFailed()<br/>(Prevents premature key deletion)
    end

    Main-->>UI: 200 OK (Signaled)

    loop While Execution Running
        Worker->>Redis: Listen for job progress/events
        opt Signal Received
            Redis-->>Worker: { kind: 'abort-job' }
            Worker->>Worker: workflow.abort()
        end
    end

    Note over Worker: Workflow finishes or interrupts

    rect rgb(5, 46, 22)
    Note over Worker: NEW: Internal Cleanup
    Worker->>Worker: Check run.status
    alt status === 'canceled'
        Worker->>Worker: NEW: throw ManualExecutionCancelledError
    end
    end

    Worker->>Redis: Bull Worker handles thrown error
    Redis->>Redis: Move job to Failed (State consistency maintained)
    Redis-->>Worker: Success (Key still exists during update)

    Note over Redis,Worker: Job lifecycle completes without "Missing key" errors

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[tomi]

What happens if this signal never reaches the worker? E.g. if worker has crashed or due to network issues. What happens to the job, will it still be cancelled?

Also where does the execution now get moved to canceled state?

[ivov]

What happens if this signal never reaches the worker? E.g. if worker has crashed or due to network issues. What happens to the job, will it still be cancelled?

Signal delivery failure is a limitation of Redis pubsub being fire and forget. This PR scopes out that failure mode but, if the worker is alive but signal is lost, the job finishes normally, whereas if the worker crashed even though the signal was sent correctly, the job is marked as failed.

Also where does the execution now get moved to canceled state?

In thw worker, workflowRun resolves with run.status === 'canceled' so the worker throws ManualExecutionCancelledError. Bull catches that error from the processor callback and moves the job to failed state, i.e. Bull's own lifecycle handles the Redis key cleanup.

[n8n-assistant]

Got released with n8n@2.15.0