Skip to content

fix(workflows): make replay journal durable - #8735

Closed
qqqys wants to merge 24 commits into
QwenLM:mainfrom
qqqys:codex/issue-8105-workflow-durable-journal
Closed

fix(workflows): make replay journal durable#8735
qqqys wants to merge 24 commits into
QwenLM:mainfrom
qqqys:codex/issue-8105-workflow-durable-journal

Conversation

@qqqys

@qqqys qqqys commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Status: Draft — scope reduction required

The target scope is single-owner durability: ordered journal writes, durable checkpoints, fail-closed artifact validation, and read-only interrupted history. Cross-process ownership/takeover and cross-process resume are deferred. The current branch still contains ownership-CAS and resume coupling, so it is not ready for review or merge until that code is removed or split into a maintainer-led follow-up. Only Critical corrections should be folded into this draft while it is being reduced.

What this PR does

This change turns workflow replay state into a durable, versioned checkpoint contract. Journal writes are serialized through a per-run queue, pause and terminal state publication wait for a durable checkpoint, and recovery validates the exact committed journal prefix while safely discarding crash-only suffix bytes. A version-aware run record distinguishes live, interrupted/recoverable, terminal, legacy display-only, and corrupted states; manifests preserve the exact script and bounded JSON args needed for replay without persisting approval or tool payloads. Persistence also fails closed across truncated data, incompatible versions, path replacement, symlinks, and hardlinks.

The text workflow history now reports an active manifest without a live owner as interrupted, including whether it is recoverable and the reason when it is not. Existing top-level terminal snapshots remain readable and continue to provide backward-compatible history.

Why it's needed

Cooperative pause/resume from the previous phase was intentionally in-memory. The old fire-and-forget journal and terminal-only snapshot could not prove which replay prefix had reached disk, so a process exit around pause or settlement could expose stale state, promote partial bytes, or silently start fresh after corruption. This PR supplies the fail-closed durability foundation required before a later phase exposes cross-process resume commands; it does not add that resume UI itself.

Reviewer Test Plan

How to verify

  1. Start a background workflow with concurrent dispatches, request pause while work is in flight, and confirm the run remains pausing until the ordered journal has flushed and the manifest has been atomically published; it should then become paused and resume without repeating the committed prefix.
  2. Restart without a live owner and inspect /workflows: an active durable record should appear as interrupted · recoverable, while missing, truncated, hash-mismatched, incompatible, or structurally invalid data should be visible as not recoverable and must never fall back to a fresh run.
  3. Cancel a paused or running workflow and exercise a timeout; each run should settle once, drain journal writes, publish a terminal record, and remain non-resumable. A cancellation that races terminal checkpoint publication must not be reported as success.
  4. Place corrupt, symlinked, hardlinked, or concurrently replaced workflow artifacts at the persistence boundary; reads and writes should fail closed without modifying an outside canary or committing bytes to a replaced inode.

Focused evidence on macOS:

  • Core durability, scheduler, runner, registry, orchestrator, and tool tests: 345 passed.
  • CLI workflow command tests: 35 passed.
  • Targeted ESLint and Prettier checks passed.
  • npm run build passed.
  • npm run typecheck passed.
  • git diff --check passed.

Evidence (Before & After)

Before: persisted workflow history only had terminal snapshots and no trustworthy marker for a paused process's committed replay prefix.

After: a run without a live owner is rendered as interrupted with recoverable or an explicit non-recoverable reason, and only a validated, atomically published checkpoint can be resumed by future entry points. No new interactive resume control is included in this PR.

Tested on

OS Status
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

Local Node.js workspace build and package-local Vitest suites; no sandbox or real-network dependency.

Risk & Scope

  • Main risk or tradeoff: this planned durability slice touches the journal writer, manifest reader/writer, runner transition barriers, storage paths, and the minimal read-only CLI consumer as one contract. It has approximately 1,471 gross production additions, which exceeds the repository's 1,000-line maintainer-awareness advisory; it is a focused fix, not a large core refactor, and the complete producer/consumer set was independently audited.
  • Not validated / out of scope: Windows and Linux were not run locally; CI should validate those platforms. Cross-process resume commands, dialog resume controls, per-agent controls, tracing, and Remote Workflow remain later or separate work.
  • Breaking changes / migration notes: the new per-run manifest uses schema version 2. Existing terminal snapshot files remain readable and display-only, corrupted or incompatible records fail closed, and no destructive migration is performed.

Linked Issues

Part of #8105

中文说明

本 PR 做了什么

本次改动把 Workflow replay 状态升级为持久、带版本的 checkpoint 契约。Journal 写入通过每个 run 独立的有序队列串行化;pause 和 terminal 状态只有在 checkpoint 持久化后才发布;恢复时严格校验已经提交的 journal 前缀,并安全丢弃仅由崩溃留下的尾部字节。新的版本感知运行记录能够区分 live、interrupted/recoverable、terminal、仅展示的 legacy 以及损坏状态;manifest 保存 replay 所需的精确脚本和有界 JSON args,但不持久化 approval 或 tool payload。对于截断数据、不兼容版本、路径替换、符号链接和硬链接,持久化逻辑都会 fail closed。

文本 Workflow 历史现在会把“存在 active manifest 但没有 live owner”的运行显示为 interrupted,同时给出是否可恢复以及不可恢复原因。现有顶层 terminal snapshot 仍然可读,并继续提供向后兼容的历史记录。

为什么需要

上一阶段的协作式 pause/resume 按计划只存在于内存中。旧的 fire-and-forget journal 和仅 terminal snapshot 无法证明哪一段 replay 前缀已经真正落盘,因此进程若在 pause 或结算附近退出,可能暴露陈旧状态、把部分字节错误升级为已提交数据,或者在数据损坏后静默 fresh run。本 PR 为后续阶段开放跨进程 resume 命令提供 fail-closed 的持久化基础;本 PR 本身不增加该 resume UI。

Reviewer 测试计划

如何验证

  1. 启动包含并发 dispatch 的后台 Workflow,在仍有工作执行时请求 pause;确认状态在有序 journal 完成 flush 且 manifest 原子发布前一直保持 pausing,之后才进入 paused,恢复时不会重复执行已提交前缀。
  2. 在没有 live owner 的新进程中查看 /workflows:active 持久记录应显示为 interrupted · recoverable;journal 缺失或截断、hash 不匹配、版本不兼容、结构损坏时,应明确显示不可恢复,且绝不能静默退化为 fresh run。
  3. 取消 paused/running Workflow,并触发 timeout;每个 run 都应只结算一次、排空 journal 写入、发布 terminal 记录并保持不可恢复。若取消与 terminal checkpoint 发布发生竞态,也不能错误报告成功。
  4. 在持久化边界预置损坏文件、符号链接、硬链接,或并发替换 artifact;读写必须 fail closed,不能修改边界外 canary,也不能把字节提交到被替换的 inode。

macOS 聚焦验证证据:

  • Core durability、scheduler、runner、registry、orchestrator 与 tool 测试:345 项通过。
  • CLI Workflow 命令测试:35 项通过。
  • 目标 ESLint 与 Prettier 检查通过。
  • npm run build 通过。
  • npm run typecheck 通过。
  • git diff --check 通过。

前后对比证据

之前:持久化 Workflow 历史只有 terminal snapshot,无法可信标识 paused 进程已经提交到磁盘的 replay 前缀。

之后:没有 live owner 的 run 会显示为 interrupted,并标注 recoverable 或明确的不可恢复原因;只有经过校验并原子发布的 checkpoint 才能被未来入口恢复。本 PR 不包含新的交互式 resume 控件。

测试平台

OS 状态
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

环境(可选)

本地 Node.js workspace build 和 package-local Vitest 测试;不依赖 sandbox 或真实网络。

风险与范围

  • 主要风险或取舍:这个计划内的 durability 切片把 journal writer、manifest reader/writer、runner transition barrier、storage path 以及最小只读 CLI consumer 作为一个契约共同修改。生产代码 gross additions 约 1,471 行,超过仓库 1,000 行的 maintainer-awareness 提醒线;它是聚焦的 fix,不是大规模 core refactor,并且全部 producer/consumer 已完成独立审计。
  • 未验证或不在范围内:Windows 和 Linux 未在本地执行,需由 CI 验证。跨进程 resume 命令、Dialog resume 控件、per-agent controls、Tracing 与 Remote Workflow 属于后续阶段或独立工作。
  • 破坏性变更或迁移说明:新的 per-run manifest 使用 schema version 2。现有 terminal snapshot 保持可读且只用于展示;损坏或不兼容记录会 fail closed;本 PR 不执行破坏性迁移。

关联 Issue

Part of #8105

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants