feat(workflows)!: add durable cross-session resume - #1509
Conversation
Add an optional DBOS-backed durable workflow backend with file/in-memory fallbacks, plus ctx.tool, UI, and stage checkpoint primitives for resumable side effects. Cache durable workflow metadata in session history and wire /workflow resume to a workflow-specific resume catalog while preserving live-run controls. Cover durable state, DBOS adapter behavior, resume catalog/runtime flows, and cross-session metadata with unit tests, and document configuration and semantics. Assistant-model: OpenAI GPT-5
Wire the lazy DBOS SDK adapter through launch, workflow control, and checkpoint hydration so fresh sessions can replay persisted state. Add durable stage/task replay, stronger resume discovery, file backend locking and merge semantics, root workflow and failure-state filters, and coverage for the DBOS hydration and resume paths.
Sync bun.lock with the optional DBOS SDK dependency and durable workflow packages. Scan Atomic custom JSONL entries for workflow resume metadata, hydrate DBOS workflow metadata without marking resumable roots completed, and serialize durable checkpoint writes with a flush barrier so write failures surface before completion. Preserve graph/store visibility when replaying cached stages and tasks, and keep no-arg /workflow resume aligned with the live run picker before falling back to durable history. Assistant-model: OpenAI Codex
Persist mutable DBOS metadata without relying on write-once helper workflow results and remove the invalid duplicationPolicy parameter from DBOS calls. Await stage checkpoint persistence, add a DBOS initialization barrier before workflow dispatch, and make no-arg workflow resume surface the durable selector while preserving live picker behavior. Differentiate repeated ctx.tool calls with ordinal checkpoint identity, make retry backoff cancellation-aware, and preserve first-run onboarding from origin/main. Assistant-model: Atomic Subagent
Add durable replay coverage for ctx.chain, ctx.parallel, and child workflow calls so composite workflow operations do not re-run after resume. Flush terminal durable workflow status before returning, only update the DBOS replay mirror after checkpoint acceptance, and keep /workflow resume durable history visible when completed local runs exist. Assistant-model: OpenAI GPT-5
Keep the prepared durable catalog available through resume so scan-only session-cache entries can be selected and resumed. Preserve structured schema-backed stage replay values, await parallel fail-fast finalizers, clean up sleepOrAbort listeners on normal completion, and prefer explicit stage replay keys before falling back to external lookup. Assistant-model: OpenAI GPT-5
Stabilize child workflow replay keys so repeated child calls resume completed work without re-executing side effects. Persist direct stages under the durable replay key, record ctx.exit terminal durable metadata, surface durable history even when live runs exist, and propagate durableBackend into child workflow runs. Assistant-model: OpenAI GPT-5
Scope child workflow internal checkpoints to the root workflow so nested side effects replay consistently across sessions. Refuse stale cache-only resume entries when no durable backend state exists, switch durable replay identities to SHA-256 digests, check cancellation after tool functions resolve before checkpointing or returning, and recover stale file-backend locks after crashes.
Suppress stale terminal cache entries when backend state is terminal so old session JSONL cannot resurrect completed workflows. Open the workflow overlay after a successful durable resume and combine live and durable entries in the no-arg /workflow resume picker. Tighten scoped checkpoint listing to exclude sibling scopes and clean up the merged run import. Assistant-model: OpenAI GPT-5 Codex
Preserve empty string stage outputs when checkpointing durable stage results. Release stage limiter slots when finalization fails, hydrate durable entries before mixed live/durable resume selection, preserve replayed stage parent/frontier graph state, and avoid opening a second picker after combined picker dismissal. Assistant-model: OpenAI GPT-5
Implementation notes (part 1/2)Implementation NotesTask: Implement GitHub issue #1498 in bastani-inc/atomic: #1498 ("Add cross-session resumability for Atomic workflows with DBOS"). Decisions and Tradeoffs1. Durable backend seam (not direct DBOS coupling)Decision: Created a
Rationale: The issue says "integrate DBOS with the workflow backend" and "do not change the frontend workflow syntax." A backend seam lets the engine persist Tradeoff: The DBOS adapter ( 2. ctx.tool primitiveDecision: Added Rationale: The issue explicitly requires "ctx.tool which allows you to run any typescript code and cache the result for DBOS." Tradeoff: The hash function ( 3. File-backed backend as default (no Postgres required)Decision: When Rationale: The issue says "Save state in DBOS by caching on session file." The file backend provides cross-process resume without infrastructure, matching the zero-setup expectation. DBOS remains available as an upgrade path. Tradeoff: File I/O is not as robust as Postgres for concurrent writes. The atomic write (temp + rename) mitigates corruption risk. 4. Session JSONL cache entriesDecision: Added Rationale: The issue requires "durable state cached on the session file" and a 5. Cancellation/failure/retry semanticsDecision: Mapped to existing workflow failure classification plus durable status tracking:
6. Mock for builtin-workflows-helpers.tsDecision: Added a Rationale: The Deviations from Research
Validation
Open Gaps / Future Work
QA E2E VideoNo QA E2E video was produced for this implementation. The change is backend/library code (durable state management, ctx.tool primitive, resume catalog) with no user-visible UI scenario that can be driven via playwright-cli. The Follow-up implementation (issue #1498, second pass)Closed the open gaps from the first implementation pass:
Key design decision: opt-in cross-session persistenceChanged the default durable backend from file-backed to in-memory (factory.ts). Rationale:
Validation
Remaining notes
Validation pass (independent review, issue #1498)Reviewer independently re-ran all validation and probed the durable-replay path end-to-end. Commands run / results
Independent end-to-end engine smoke (reviewer-authored)Wrote a throwaway
Observations / minor flags (non-blocking)
QA E2E videoNo QA E2E video produced. Rationale: this is backend/library + TUI command-path code with no user-visible browser UI scenario; Safety to commitSafe to commit. No fixes required from this review. The only recommendation is reverting the incidental root Files touched (this pass)
Review-fixes validation pass (issue #1498)Independent validation of the staged review-fixes (on top of commit Commands / results
Review findings — all addressed
Non-blocking concern: DBOS read-side hydrationThe TUI/tmux E2ENot practical: CLI not built (no dist/; requires tsgo + asset copy), DBOS SDK not installed (minimumReleaseAge gate). Narrower proof: overlay integration test (12/12) exercises the QA E2E videoNo QA E2E video. Backend/library + TUI command-path code; no browser UI scenario; interactive TUI blocked by unbuilt binary. Executable proof is the test suite (2620/2620 unit + 12/12 overlay + 71/71 durable). Safety to commitSafe to commit. No fixes required. Findings written to Commit
Final independent validation after DBOS hydration follow-upValidation performed from Commands run:
Tiny validation fix made: corrected Final acceptance assessment: no known unaddressed acceptance gaps remain in the implementation diff. DBOS-backed resume remains lazily initialized and mock/integration-tested without requiring a live Postgres instance in this validation environment; the real DBOS path now has read-side hydration APIs and resume/list preparation hooks. QA E2E video note: no browser QA video applies. This is backend/library plus terminal/TUI command-path behavior. Safe-to-commit verdict: safe to commit after including the tiny docs correction. No final blockers identified. Review-fix implementation passAddressed the post-review blockers called out in the refreshed research:
Review-fix validation
Remaining limitationDBOS checkpoint storage is now wired to real SDK lifecycle/control APIs where feasible without making DBOS a required dependency. Atomic still executes workflow code itself and mirrors checkpoints locally for synchronous replay queries; DBOS is used for durable workflow/control records and checkpoint-workflow records. A full DBOS-native execution model for arbitrary dynamically discovered Atomic workflow definitions would require a broader architecture change and a live Postgres-backed integration environment. Review-gap preflight (delegated analysis)
Follow-up: DBOS read-side hydration fixImplemented the remaining DBOS read-side hydration gap found by validation:
Validation for DBOS hydration fix
Remaining limitationThe real DBOS SDK handle stores Atomic checkpoints as separate Final commit after review fixes
Final latest validation after risk fixesFinal validation was run from Risk verification:
Commands / results:
QA E2E note: Safe-to-commit verdict: safe to commit. No blockers found. Follow-up latest risk fixesAddressed the two validation risks found after the latest production-readiness pass:
Validation for this follow-up:
Remaining note: the DBOS failure-propagation test intentionally exercises a failing mock backend and emits the expected Latest production-fix passImplemented the latest research findings after commit
Validation for latest pass:
No QA E2E video was produced: this pass is backend/library plus terminal/TUI command-path behavior. The executable proof is the overlay command integration test plus durable unit tests and full unit suite. Latest preflight / review findings after commit 800c734Performed a read-only preflight from Evidence gathered:
Recommended targeted validation after fixes:
Latest final validation pass (delegated, issue #1498)Validation performed from Diff review findingsConfirmed addressed by code/tests:
Potential remaining blockers / risks found by manual review:
Commands run / results
TUI / QA E2E assessment
Safe-to-commit verdictNot fully safe to commit as "complete" if the latest acceptance checklist requires every DBOS checkpoint write to be awaited and cached Latest final commit
Current preflight after commit 37c34a0Preflight-only pass from Evidence gathered:
Current unresolved findings confirmed:
Recommended targeted validation after fixes:
Current unresolved findings implementation passImplemented the latest unresolved #1498 fixes from the current research/preflight pass:
Validation run in this pass (partial so far):
Final validation for current unresolved findings pass:
QA E2E video note remains unchanged: no browser UI applies. Full interactive TUI E2E remains impractical without a configured Atomic interactive session and model/runtime credentials; headless overlay integration tests exercise the Current independent validation after unresolved-finding fixesValidation performed from Diff review findings:
Commands / results:
QA / E2E note: Safe-to-commit verdict: safe to commit. No blockers found in this validation pass. Current final commit
Final preflight after commit d4f32a4Preflight-only pass from Evidence gathered:
Current unresolved findings confirmed:
Earlier resolved findings appear covered by existing tests: lockfile, DBOS versioned metadata, invalid duplicate params removal, custom JSONL cache shape, file concurrency, root/non-recoverable filtering, UI/tool ordinal identity, retry cancellation, DBOS activation/hydration, and onboarding preservation. Recommended validation after fixes is listed in Final workspace recheck note for preflight: after writing Final unresolved findings implementation passImplemented the latest issue #1498 review findings after commit
|
Implementation notes (part 2/2)
Validation for this pass:
QA E2E video note remains unchanged: no browser UI applies. Full interactive TUI E2E remains impractical without a configured Atomic interactive session and model/runtime credentials; headless overlay integration tests cover the Final validation after newest durable replay fixesValidation performed from Diff review findings:
Commands / results:
QA / E2E note: Safe-to-commit verdict: safe to commit. No blockers found in this validation pass. Final durable composite replay commit
Scan-only resume validation passFinal independent validation after the scan-only resume/latest issue #1498 fixes. No code changes were made in this validation pass. Findings were written to Diff review verified:
Commands / results:
QA / E2E note: Safe-to-commit verdict: safe to commit. No blockers found. Scan/session-cache resume preflightPreflight-only pass from Evidence gathered:
Current unresolved findings confirmed:
Recommended validation after fixes:
Scan-only resume and structured stage replay fixesImplemented the latest issue #1498 unresolved findings:
Validation:
QA E2E video note: no browser UI applies. Full interactive Atomic TUI E2E remains impractical without a configured Atomic interactive session, workflow fixtures, and model/runtime credentials; the headless overlay integration suite covers the Final independent validation after child workflow/ctx.exit/resume/backend propagation fixesValidation performed from Diff review — all five findings verified as resolved
Earlier regression areas confirmed covered by existing tests. Commands / results
TUI / QA E2E
Safe-to-commit verdictSafe to commit. No blockers found. Findings written to Scan-only durable resume commit
Child-workflow and ctx.exit preflight (issue #1498)Preflight-only pass after commit Evidence gathered:
Current unresolved findings confirmed (all five, with exact code locations):
None of these five findings are covered by existing regression tests (grep confirmed). Recommended validation after fixes is listed in Child workflow / ctx.exit commit
Root-child & replay-robustness fixes (latest pass)Implemented the latest issue #1498 unresolved findings:
Refactor: extracted durable terminal-status finalization from Validation
QA E2E video: no browser UI scenario applies (backend/library + TUI command-path). Full interactive TUI E2E remains impractical without a configured Atomic session/model runtime; the executable proof is the durable + overlay test suites including the Root-child checkpoint & durable robustness preflight (issue #1498)Preflight-only pass after commit Baseline at HEAD (all clean):
Current unresolved findings confirmed (all five, with exact code locations):
Recommended validation after fixes is listed in Earlier resolved findings remain covered by existing regression tests. Root-child / digest / stale validation pass (issue #1498)Final independent validation after the latest root-child/digest/stale fixes (uncommitted diff on top of HEAD Diff review — all five findings verified as resolved
Earlier regression areas remain covered by the full durable + overlay + slash-dispatch + onboarding suites. Commands / results
TUI / QA E2E
Safe-to-commit verdictSafe to commit. No blockers found. Root-child durable checkpoint commit
Terminal-cache / overlay / latest preflight (issue #1498)Preflight-only pass after commit Baseline at HEAD (all clean):
Current unresolved findings confirmed (all five, with exact code locations):
Recommended validation after fixes is listed in Earlier resolved findings remain covered by existing regression tests. Terminal-cache / overlay / scoped-listing validation pass (issue #1498)Final independent validation after the terminal-cache/overlay/scoped-listing fixes (uncommitted diff on top of HEAD Diff review — all five findings verified as resolved
Earlier regression areas remain covered by the full durable + overlay + slash-dispatch + onboarding suites. Commands / results
TUI / QA E2E
Safe-to-commit verdictSafe to commit. No blockers found. Terminal-cache / durable overlay commit
Stage/frontier/selector preflight (issue #1498)Preflight-only pass after commit Baseline at HEAD (all clean):
Current unresolved findings confirmed (all five, with exact code locations):
Earlier resolved findings remain covered by the durable + overlay + slash-dispatch + onboarding suites. Recommended validation after fixes is listed in Stage/frontier/selector validation pass (issue #1498)Final independent validation after the stage/frontier/selector fixes (uncommitted diff on top of HEAD Diff review — all five findings verified as resolved
Earlier regression areas remain covered: durable-resume-runtime/root-child/dbos-backend/stage-primitive (80 pass), overlay (19 tests), slash-dispatch resume (10 pass), onboarding (39 pass). Commands / results
TUI / QA E2E
Safe-to-commit verdictSafe to commit. No blockers found. Stage/frontier durable replay commit
|
|
PR Review: feat(workflows): add durable cross-session resume Reviewed all 14 files under Findings grouped by severity — most are scaling/edge-cases rather than correctness blockers. Potential bugs / correctness
Performance / scaling
Robustness / minor
Security / privacy
Design note (not a blocker) The replay model relies on deterministic call ordering — Overall: solid, cohesive implementation with strong tests and good adherence to repo conventions (file-length gate respected, raw-TS no-build-step preserved, optional DBOS dep correctly under |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Assistant-model: GPT-5.5
PR Review — Durable cross-session workflow resume (#1509)Thorough, well-documented change. The backend seam is cleanly abstracted, checkpoints are idempotent, cancellation is re-checked at the Performance / scalability
Maintainability
Minor
Nothing here blocks correctness for the documented opt-in paths — items 1–3 are the ones I'd most want addressed (or explicitly deferred with a tracking note) since they affect behavior under real load. |
Assistant-model: GPT-5.5
PR Review:
|
…rkflow-resume # Conflicts: # packages/workflows/CHANGELOG.md
|
Review: durable cross-session workflow resume (issue 1509) Reviewed the durable backend system, resume catalog/runtime, and the engine/TUI wiring. This is a well-architected feature: the backend seam is clean, the in-memory/file/DBOS implementations share an honest interface, the changelog and Findings grouped by severity. Most are performance/scalability concerns rather than correctness blockers. Performance / scalability (the main theme)
Correctness — worth a second look
Minor / nits
Security / data handling No credential leakage spotted — Tests Coverage is strong and maps clearly to the fixes. Two gaps to consider: a test pinning the schema-stage non-empty-string behavior (item 5), and a file-backend concurrency/contention test exercising the lock path. Overall: solid, careful implementation. None of the above is a hard blocker, but the file-backend write amplification (1) + event-loop-blocking lock (2) and the unbounded DBOS metadata (3) are the ones I would most want addressed before this sees heavy real-world use. Automated review by Claude (Opus 4.8) |
Default workflow durability now uses the lock-protected file backend under ~/.atomic/workflow-durable so cross-session /workflow resume works without opt-in environment variables. DBOS remains an optional upgrade when DBOS_SYSTEM_DATABASE_URL is configured, with file-backed durability as the fallback. BREAKING CHANGE: workflow durability no longer defaults to a process-local in-memory backend and the ATOMIC_WORKFLOW_DURABLE_DIR opt-in path is removed; in-memory durability is only for explicit test/custom backend overrides. Assistant-model: GPT-5.5
claude
Bot
changed the title
|
Review: durable cross-session workflow resume (issue 1498) Thorough, well-structured PR. The backend seam is a clean abstraction, the durable subsystem is nicely modularized (every file under the 500-line gate), typing is strict with no any/unknown slop, the changelog is detailed, and test coverage is substantial (10 durable unit files + integration suites). The reasoning around scoped child checkpoints, post-resolution cancellation re-checks, and the empty-string-vs-status-object distinction is excellent. Findings below, ordered by impact. 🔴 Performance — the default file backend blocks the event loop (highest impact) This commit makes
Suggestions: use the already-written per-workflow 🟠 Dead / contradictory code
🟠 DBOS metadata grows unbounded
🟡 Smaller correctness/robustness notes
Nits
Verification I could not run Overall strong, careful work — the main ask before merge is the default-backend event-loop blocking and single-shared-file write amplification, since the breaking change puts that path in front of all users. 🤖 Generated with Claude Code |
Resolve the failing CI test and harden the durable workflow resume feature landed in #1498. CI fix (the failing `executor.run — lifecycle persistence` test): - The `workflow.durable.checkpoint` cache-entry append is gated on a *persistent* durable backend. Several test files mutate the global durable backend singleton (bun shares process state across files) and never reset it, leaking an in-memory backend that made this assertion flaky. Every test that calls `setDurableBackend(...)` now resets it in `afterEach`, and the lifecycle test resets the singleton in `beforeAll`/`afterAll` for defense in depth. Durable resume correctness: - A durable `running` handle may belong to a crashed process, so it stays resumable at the backend/catalog level (cross-session crash recovery). Same- session double-resume is prevented session-aware: the selector hides and resume refuses only when there is an *actively-executing live run* in this session, with an intuitive error pointing at `/workflow connect`/`/workflow kill`. - In-progress LM stage-session checkpoints no longer collide with completed stage-output checkpoints (separate backend indexes), DBOS records persist by checkpoint id, stale quit snapshots are removed before reusing the workflow id, and mid-session LM resume sends `Continue` instead of re-sending the original prompt — fixing repeated quit/resume cycles that emptied chats. - Removed an unnecessary `replayKey` override in the executor stage factory that broke continuation-replay topology validation. Quit vs kill UX: - Orchestrator/CLI `q` is now a resumable quit/detach (durable handle → `paused`), not a kill. Only `/workflow kill` authoritatively cancels a workflow. The background widget/status list render a `quit` badge with a "resumable via /workflow resume" note. Tests: 2716 unit + 247 integration passing; typecheck, lint, file-length clean. Refs #1498
Review: durable cross-session workflow resume (#1509)Reviewed the durable backend system, resume catalog/runtime, and the 🔴 Performance — default file backend rewrites all state on every checkpoint
Consequences now that durability is on by default:
You already wrote Minor but related: the comment in 🟠 Stale-lock reclaim has a TOCTOU race
🟠 Privacy/security — sensitive data persisted to disk by defaultDurability is now always on and writes workflow inputs,
🟡 Replay identity has no workflow-version guardTool/UI/stage replay keys are 🟡 Minor
Test coverageStrong — 11 durable unit files plus integration coverage for the overlay/resume dispatch paths. If you address the file-backend points above, please add a test for (a) the shared-file growth/pruning behavior and (b) concurrent-process lock contention / stale-lock reclaim, since those are the riskiest paths and currently the least exercised. Overall this is solid, careful work. The performance of the default file backend is the one item I'd consider a merge blocker given it's now always-on; the rest are hardening/correctness follow-ups. 🤖 Generated with Claude Code |
Store default durable workflow state in per-workflow files, prune terminal file-backed state, and avoid /tmp fallback when no home directory is available. Add restrictive file permissions, owner-marked stale-lock reclaim, a documented in-memory privacy opt-out, and coverage for default backend persistence. Assistant-model: GPT-5.5
Review:
|
Avoid per-workflow backend all-file scans on point lookups, document plaintext durability retention, and make file permission assertions portable on Windows. Keep reset scoped to workflow state files and remove scratch durable-resume QA workflows from the project workflow catalog. Assistant-model: GPT-5.5
|
Review: feat(workflows)! add durable cross-session resume (PR 1509) Reviewed the durable backend system, primitives, resume catalog/runtime, and the file/DBOS adapters. This is a large, well-structured change — the backend seam is a clean abstraction, the changelog and Feedback below, roughly ordered by impact. Nothing here is a correctness blocker; the top item is a performance concern worth weighing before this lands as an always-on default. 1. Performance — synchronous full-file rewrite on every checkpoint (now default-on). 2. Atomics.wait blocks the event loop during lock contention. 3. Unbounded metadata-step growth in the DBOS backend. 4. isDbosDuplicateWorkflowError matches on a loose message regex. 5. wrapSchemaStageForDurability silently skips non-empty string results. 6. Minor / nits.
Strengths worth calling out.
Overall this looks solid and carefully tested. The main thing I would want resolved before shipping as always-on is the per-checkpoint synchronous-write cost (items 1 and 2) under realistic workflow sizes. |
* feat(workflows): add durable cross-session resume Add an optional DBOS-backed durable workflow backend with file/in-memory fallbacks, plus ctx.tool, UI, and stage checkpoint primitives for resumable side effects. Cache durable workflow metadata in session history and wire /workflow resume to a workflow-specific resume catalog while preserving live-run controls. Cover durable state, DBOS adapter behavior, resume catalog/runtime flows, and cross-session metadata with unit tests, and document configuration and semantics. Assistant-model: OpenAI GPT-5 * fix(workflows): complete durable DBOS resume wiring Wire the lazy DBOS SDK adapter through launch, workflow control, and checkpoint hydration so fresh sessions can replay persisted state. Add durable stage/task replay, stronger resume discovery, file backend locking and merge semantics, root workflow and failure-state filters, and coverage for the DBOS hydration and resume paths. * fix(workflows): harden durable resume persistence Sync bun.lock with the optional DBOS SDK dependency and durable workflow packages. Scan Atomic custom JSONL entries for workflow resume metadata, hydrate DBOS workflow metadata without marking resumable roots completed, and serialize durable checkpoint writes with a flush barrier so write failures surface before completion. Preserve graph/store visibility when replaying cached stages and tasks, and keep no-arg /workflow resume aligned with the live run picker before falling back to durable history. Assistant-model: OpenAI Codex * fix(workflows): finalize DBOS resume semantics Persist mutable DBOS metadata without relying on write-once helper workflow results and remove the invalid duplicationPolicy parameter from DBOS calls. Await stage checkpoint persistence, add a DBOS initialization barrier before workflow dispatch, and make no-arg workflow resume surface the durable selector while preserving live picker behavior. Differentiate repeated ctx.tool calls with ordinal checkpoint identity, make retry backoff cancellation-aware, and preserve first-run onboarding from origin/main. Assistant-model: Atomic Subagent * fix(workflows): complete durable composite replay Add durable replay coverage for ctx.chain, ctx.parallel, and child workflow calls so composite workflow operations do not re-run after resume. Flush terminal durable workflow status before returning, only update the DBOS replay mirror after checkpoint acceptance, and keep /workflow resume durable history visible when completed local runs exist. Assistant-model: OpenAI GPT-5 * fix(workflows): preserve scan-only durable resume state Keep the prepared durable catalog available through resume so scan-only session-cache entries can be selected and resumed. Preserve structured schema-backed stage replay values, await parallel fail-fast finalizers, clean up sleepOrAbort listeners on normal completion, and prefer explicit stage replay keys before falling back to external lookup. Assistant-model: OpenAI GPT-5 * fix(workflows): persist child and exit resume state Stabilize child workflow replay keys so repeated child calls resume completed work without re-executing side effects. Persist direct stages under the durable replay key, record ctx.exit terminal durable metadata, surface durable history even when live runs exist, and propagate durableBackend into child workflow runs. Assistant-model: OpenAI GPT-5 * fix(workflows): scope child durable checkpoints to roots Scope child workflow internal checkpoints to the root workflow so nested side effects replay consistently across sessions. Refuse stale cache-only resume entries when no durable backend state exists, switch durable replay identities to SHA-256 digests, check cancellation after tool functions resolve before checkpointing or returning, and recover stale file-backend locks after crashes. * fix(workflows): connect durable resume selections Suppress stale terminal cache entries when backend state is terminal so old session JSONL cannot resurrect completed workflows. Open the workflow overlay after a successful durable resume and combine live and durable entries in the no-arg /workflow resume picker. Tighten scoped checkpoint listing to exclude sibling scopes and clean up the merged run import. Assistant-model: OpenAI GPT-5 Codex * fix(workflows): preserve replayed stage graph state Preserve empty string stage outputs when checkpointing durable stage results. Release stage limiter slots when finalization fails, hydrate durable entries before mixed live/durable resume selection, preserve replayed stage parent/frontier graph state, and avoid opening a second picker after combined picker dismissal. Assistant-model: OpenAI GPT-5 * fix(workflows): harden durable resume review blockers Assistant-model: GPT-5.5 * fix(workflows): avoid headless durable resume picker Assistant-model: GPT-5.5 * feat(workflows)!: enable durable resume by default Default workflow durability now uses the lock-protected file backend under ~/.atomic/workflow-durable so cross-session /workflow resume works without opt-in environment variables. DBOS remains an optional upgrade when DBOS_SYSTEM_DATABASE_URL is configured, with file-backed durability as the fallback. BREAKING CHANGE: workflow durability no longer defaults to a process-local in-memory backend and the ATOMIC_WORKFLOW_DURABLE_DIR opt-in path is removed; in-memory durability is only for explicit test/custom backend overrides. Assistant-model: GPT-5.5 * fix(workflows): stabilize durable resume, quit/kill UX, and CI isolation Resolve the failing CI test and harden the durable workflow resume feature landed in #1498. CI fix (the failing `executor.run — lifecycle persistence` test): - The `workflow.durable.checkpoint` cache-entry append is gated on a *persistent* durable backend. Several test files mutate the global durable backend singleton (bun shares process state across files) and never reset it, leaking an in-memory backend that made this assertion flaky. Every test that calls `setDurableBackend(...)` now resets it in `afterEach`, and the lifecycle test resets the singleton in `beforeAll`/`afterAll` for defense in depth. Durable resume correctness: - A durable `running` handle may belong to a crashed process, so it stays resumable at the backend/catalog level (cross-session crash recovery). Same- session double-resume is prevented session-aware: the selector hides and resume refuses only when there is an *actively-executing live run* in this session, with an intuitive error pointing at `/workflow connect`/`/workflow kill`. - In-progress LM stage-session checkpoints no longer collide with completed stage-output checkpoints (separate backend indexes), DBOS records persist by checkpoint id, stale quit snapshots are removed before reusing the workflow id, and mid-session LM resume sends `Continue` instead of re-sending the original prompt — fixing repeated quit/resume cycles that emptied chats. - Removed an unnecessary `replayKey` override in the executor stage factory that broke continuation-replay topology validation. Quit vs kill UX: - Orchestrator/CLI `q` is now a resumable quit/detach (durable handle → `paused`), not a kill. Only `/workflow kill` authoritatively cancels a workflow. The background widget/status list render a `quit` badge with a "resumable via /workflow resume" note. Tests: 2716 unit + 247 integration passing; typecheck, lint, file-length clean. Refs #1498 * fix(workflows): harden file-backed durability Store default durable workflow state in per-workflow files, prune terminal file-backed state, and avoid /tmp fallback when no home directory is available. Add restrictive file permissions, owner-marked stale-lock reclaim, a documented in-memory privacy opt-out, and coverage for default backend persistence. Assistant-model: GPT-5.5 * fix(workflows): address durable backend review feedback Avoid per-workflow backend all-file scans on point lookups, document plaintext durability retention, and make file permission assertions portable on Windows. Keep reset scoped to workflow state files and remove scratch durable-resume QA workflows from the project workflow catalog. Assistant-model: GPT-5.5
1 participant
Summary
Introduces a pluggable durable workflow backend that persists
ctx.*operation checkpoints across sessions, enabling a new Atomic session to resume a workflow from the last completed checkpoint — without re-running completed work or changing workflow authoring syntax. File-backed durability is now on by default (rooted at~/.atomic/workflow-durable). Closes #1498.Breaking Changes
~/.atomic/workflow-durable. The previous process-local in-memory default andATOMIC_WORKFLOW_DURABLE_DIRopt-in are removed. In-memory durability is available only as an explicit test/custom backend override, or via theATOMIC_WORKFLOW_DURABLE=0/false/off/memoryprivacy opt-out.Key Changes
Durable Backend System (
packages/workflows/src/durable/)~/.atomic/workflow-durable), DBOS-backed (Postgres viaDBOS_SYSTEM_DATABASE_URL, lazily initialized), and in-memory (test/custom override only).0700/0600); stale lock directories from crashed processes are reclaimed automatically.ctx.toolprimitive: durable cached tool execution with optional exponential-backoff retry (retriesAllowed,maxAttempts,intervalMs,backoffRate). Completed side effects are never re-run on resume.ctx.uicheckpointing: prompt responses cached by stable identity (method + label/message + options + call order) so resumed workflows skip already-answered prompts.ctx.stage/ctx.task/ctx.chain/ctx.parallel/ childctx.workflowcheckpointing: stage outputs recorded with stable ordinal replay keys; child workflow calls checkpoint the completed result at the parent boundary.ctx.*side effects are keyed under the root boundary so an interrupted child does not re-execute completed work on parent resume.durableHash) replacing the prior 32-bit DJB2 hash to eliminate collisions across distinct tool/stage identities.completed/failed/cancelled) is flushed before the run returns; cancellation races are guarded atctx.tooland retry boundaries.Resume Catalog & Runtime
scanResumableWorkflows: scans session JSONL files forworkflow.durable.checkpointentries to build the/workflow resumeselector without querying the backend.DBOS_SYSTEM_DATABASE_URL,/workflow resumehydrates the in-memory mirror from Postgres so prior-session workflows are discoverable.runninghandles from crashed processes remain resumable at the catalog level; same-session double-resume is blocked only when an actively-executing live run exists in the current session.Quit vs Kill UX
q(orchestrator/CLI) now pauses a workflow (durable handle →paused) rather than killing it. Background widget and status list render aquitbadge with a "resumable via/workflow resume" hint./workflow killonly cancels: kill is now the only path that authoritatively cancels a workflow, preventing accidental data loss during quit/resume cycles./workflow resumeTUI Integration/workflow resumeopens a unified selector showing live runs alongside cross-session durable workflows; dismissing returns to chat without a second prompt.stalecache-only entries are refused rather than silently re-run from scratch.@dbos-inc/dbos-sdkoptional dependency added to@bastani/atomicso DBOS-backed execution is available without a separate install; the adapter is loaded lazily and the workflows package remains dependency-free.Bug Fixes
ScopedDurableBackend.listCheckpointsleaking sibling child-scope checkpoints sharing a common id prefix.ctx.ui.customreplaying cached void/undefinedresponses instead of treating them as cache misses.Continueinstead of re-sending the original prompt, preventing repeated quit/resume cycles from emptying chats.replayKeyoverride in the executor stage factory that broke continuation-replay topology validation.afterEach/afterAll.Tests & Docs
/workflow resumedispatch; added regression suite for overlay resume edge cases; addeddurable-stage-session-resumeunit coverage.packages/coding-agent/docs/workflows.mdwith durable resume configuration, backend selection,ctx.toolauthoring guidance, and quit/kill UX.Configuration
~/.atomic/workflow-durableDBOS_SYSTEM_DATABASE_URL=<url>ATOMIC_WORKFLOW_DURABLE=0/false/off/memoryValidation
QA
No QA E2E video applies: this is backend/library plus terminal/TUI command-path behavior, not a browser UI flow. The executable proof is the headless overlay command integration suite plus the durable unit coverage above.