Skip to content

fix(engine): isolate operation completion from best-effort side-effects - #2823

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/worker-completion-decouple
Jul 20, 2026
Merged

fix(engine): isolate operation completion from best-effort side-effects#2823
nicoloboschi merged 1 commit into
mainfrom
fix/worker-completion-decouple

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #2608 that fixes the root cause of #2601 in the engine, rather than only backstopping the symptom in the poller.

execute_task marks an operation completed via _mark_operation_completed (most task types) or _mark_operation_completed_and_fire_webhook (consolidation). Both wrapped the status = 'completed' commit in one transaction with fallible side-effects — the webhook outbox insert and parent aggregation — and then swallowed every exception. So any hiccup in those side-effects rolled the completion back with them and dropped the error, leaving the operation stuck in processing forever while the log already said the work was done.

#2608 added a poller-side completion backstop that unsticks the row after the executor returns, but it fires no webhook — so on that path the consolidation webhook was silently lost, and the engine's own completion UPDATE had no status guard, so the two completion writers could both run parent aggregation.

Changes

  • Idempotency guard — every completion UPDATE in the engine is now guarded on status = 'processing' (matching fix(worker): complete successful operations #2608's poller). Whichever writer runs second sees a non-processing row, updates nothing, and does not re-run parent aggregation.
  • Graceful fallback instead of swallow — if the atomic outbox transaction fails, the completion is no longer lost with it. We commit the terminal state on its own, then deliver the consolidation webhook best-effort via the existing non-transactional _fire_consolidation_webhook. The happy path keeps the full transactional-outbox at-least-once guarantee; the failure path degrades to "completed + delivered" instead of "stuck + lost". The best-effort fire only runs when the fallback actually transitioned the row, so there is no duplicate delivery.

Testing

  • New tests/test_operation_completion.py — fast, DB-free unit tests (fake asyncpg-style connections, same style as fix(worker): complete successful operations #2608's test_worker.py):
  • uv run pytest tests/test_operation_completion.py — 3 passed
  • uv run ruff check / ruff format --check / ty check — clean on changed files

Fixes #2601.

execute_task completes an operation via _mark_operation_completed /
_mark_operation_completed_and_fire_webhook, both of which wrapped the
status='completed' commit in one transaction with fallible side-effects
(webhook outbox insert, parent aggregation) and swallowed every exception.
A hiccup in either rolled the completion back and dropped the error, leaving
the operation stuck in 'processing' forever while the log already said the
work was done (#2601). PR #2608 added a poller-side backstop that unstuck
the row but silently lost the consolidation webhook.

- On failure of the atomic outbox transaction, fall back to a completion-only
  commit and fire the consolidation webhook best-effort (non-transactional)
  instead of losing both. Happy path keeps the transactional-outbox guarantee;
  the failure path degrades to completed + delivered rather than stuck + lost.
  The best-effort fire only runs when the fallback actually transitioned the
  row, so there is no duplicate delivery.
- Guard every completion UPDATE on `status NOT IN ('completed','failed',
  'cancelled')` so an already-terminal row is never re-terminalized: keeps the
  engine idempotent with the poller backstop (#2608) and avoids double parent
  aggregation, while still completing pending/processing rows.

Adds fast DB-free regression tests (fake connections) covering the happy
path (no double-fire), the webhook-failure fallback, and the terminal-row
no-op guard.
@nicoloboschi
nicoloboschi force-pushed the fix/worker-completion-decouple branch from d0c6494 to 3a038e7 Compare July 20, 2026 12:15
@nicoloboschi
nicoloboschi merged commit 946a80b into main Jul 20, 2026
99 of 100 checks passed
@nicoloboschi
nicoloboschi deleted the fix/worker-completion-decouple branch July 20, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

__mark_completed() is never called after successful task execution

1 participant