fix(workflows): make replay journal durable - #8735
Closed
qqqys wants to merge 24 commits into
Closed
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This change turns workflow replay state into a durable, versioned checkpoint contract. Journal writes are serialized through a per-run queue, pause and terminal state publication wait for a durable checkpoint, and recovery validates the exact committed journal prefix while safely discarding crash-only suffix bytes. A version-aware run record distinguishes live, interrupted/recoverable, terminal, legacy display-only, and corrupted states; manifests preserve the exact script and bounded JSON args needed for replay without persisting approval or tool payloads. Persistence also fails closed across truncated data, incompatible versions, path replacement, symlinks, and hardlinks.
The text workflow history now reports an active manifest without a live owner as interrupted, including whether it is recoverable and the reason when it is not. Existing top-level terminal snapshots remain readable and continue to provide backward-compatible history.
Why it's needed
Cooperative pause/resume from the previous phase was intentionally in-memory. The old fire-and-forget journal and terminal-only snapshot could not prove which replay prefix had reached disk, so a process exit around pause or settlement could expose stale state, promote partial bytes, or silently start fresh after corruption. This PR supplies the fail-closed durability foundation required before a later phase exposes cross-process resume commands; it does not add that resume UI itself.
Reviewer Test Plan
How to verify
pausinguntil the ordered journal has flushed and the manifest has been atomically published; it should then becomepausedand resume without repeating the committed prefix./workflows: an active durable record should appear asinterrupted · recoverable, while missing, truncated, hash-mismatched, incompatible, or structurally invalid data should be visible as not recoverable and must never fall back to a fresh run.Focused evidence on macOS:
npm run buildpassed.npm run typecheckpassed.git diff --checkpassed.Evidence (Before & After)
Before: persisted workflow history only had terminal snapshots and no trustworthy marker for a paused process's committed replay prefix.
After: a run without a live owner is rendered as
interruptedwithrecoverableor an explicit non-recoverable reason, and only a validated, atomically published checkpoint can be resumed by future entry points. No new interactive resume control is included in this PR.Tested on
Environment (optional)
Local Node.js workspace build and package-local Vitest suites; no sandbox or real-network dependency.
Risk & Scope
fix, not a large corerefactor, and the complete producer/consumer set was independently audited.Linked Issues
Part of #8105
中文说明
本 PR 做了什么
本次改动把 Workflow replay 状态升级为持久、带版本的 checkpoint 契约。Journal 写入通过每个 run 独立的有序队列串行化;pause 和 terminal 状态只有在 checkpoint 持久化后才发布;恢复时严格校验已经提交的 journal 前缀,并安全丢弃仅由崩溃留下的尾部字节。新的版本感知运行记录能够区分 live、interrupted/recoverable、terminal、仅展示的 legacy 以及损坏状态;manifest 保存 replay 所需的精确脚本和有界 JSON args,但不持久化 approval 或 tool payload。对于截断数据、不兼容版本、路径替换、符号链接和硬链接,持久化逻辑都会 fail closed。
文本 Workflow 历史现在会把“存在 active manifest 但没有 live owner”的运行显示为 interrupted,同时给出是否可恢复以及不可恢复原因。现有顶层 terminal snapshot 仍然可读,并继续提供向后兼容的历史记录。
为什么需要
上一阶段的协作式 pause/resume 按计划只存在于内存中。旧的 fire-and-forget journal 和仅 terminal snapshot 无法证明哪一段 replay 前缀已经真正落盘,因此进程若在 pause 或结算附近退出,可能暴露陈旧状态、把部分字节错误升级为已提交数据,或者在数据损坏后静默 fresh run。本 PR 为后续阶段开放跨进程 resume 命令提供 fail-closed 的持久化基础;本 PR 本身不增加该 resume UI。
Reviewer 测试计划
如何验证
pausing,之后才进入paused,恢复时不会重复执行已提交前缀。/workflows:active 持久记录应显示为interrupted · recoverable;journal 缺失或截断、hash 不匹配、版本不兼容、结构损坏时,应明确显示不可恢复,且绝不能静默退化为 fresh run。macOS 聚焦验证证据:
npm run build通过。npm run typecheck通过。git diff --check通过。前后对比证据
之前:持久化 Workflow 历史只有 terminal snapshot,无法可信标识 paused 进程已经提交到磁盘的 replay 前缀。
之后:没有 live owner 的 run 会显示为
interrupted,并标注recoverable或明确的不可恢复原因;只有经过校验并原子发布的 checkpoint 才能被未来入口恢复。本 PR 不包含新的交互式 resume 控件。测试平台
环境(可选)
本地 Node.js workspace build 和 package-local Vitest 测试;不依赖 sandbox 或真实网络。
风险与范围
fix,不是大规模 corerefactor,并且全部 producer/consumer 已完成独立审计。关联 Issue
Part of #8105