fix(goal): count catalog previews in the unit their budget is written in - #9835
Conversation
Evidence catalog previews were cut to 240 characters while the budget they feed is 24,000 bytes. In UTF-8 those units differ by up to four times, so the guard held only for ASCII. A legal 32-claim checkpoint of Chinese claims serialized to roughly 29kB — over the cap on its own, before a single new record had been scanned — which marked the window truncated, and `truncated` switched compaction off. The one state compaction exists to resolve was the one state it refused to run in, so the Goal was stopped as `usage_limited`, the only status the reducer refuses to resume, with nothing left to salvage. An English Goal never reached that state; a Chinese one could not avoid it. Previews are now capped to 240 UTF-8 bytes on a code point boundary at the two points a catalog entry is built. Nothing changes for ASCII, where the two units already agreed; a CJK preview is shorter than before, which is the cost of the cap actually holding. With it, a full checkpoint is bounded well inside the catalog budget for every script, so a window can no longer start out truncated. A truncated window now compresses rather than stopping the Goal. Overflow means the budget is full and the newest evidence that did fit is exactly what a checkpoint folds into claims; the older evidence left behind is already covered by the previous checkpoint. Only a window that captured nothing at all has nothing to salvage, and that is the sole remaining path to `usage_limited` here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
|
Thanks for the PR! Template looks good ✓ Problem: observed bug, and I verified the mechanism in the base tree. Catalog previews are cut with Direction: in scope — a correctness fix inside the Goal evidence runtime, no product-direction questions. Size: core paths ( Approach: the scope feels right, and I checked whether the byte-cap half alone would do. It fixes the reported CJK deadlock, but not the entry-count truncation path ( Risk: no elevated risk signals — neither changed file matches the revert-correlated paths. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到的 bug，且机制已在基线代码中核实。catalog preview 用 方向：在范围内——Goal evidence 运行时内部的正确性修复，无产品方向问题。 规模：核心路径（ 方案：范围合理，且核实过"只做字节封顶"是否足够。字节能修复报告的 CJK 死锁，但条目数截断路径（ 风险：无升级风险信号——改动文件均未命中与 revert 相关的路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewRead against What I verified:
One honest reservation, non-blocking: the PR description says the older evidence left behind by a salvaged checkpoint "is already covered by the previous checkpoint's claims" — strictly, those dropped records are post-cursor, so they are not in the previous claims, and once the salvaged checkpoint lands, the cursor moves past them and they lose citability. That is still a strictly better outcome than the old behavior (unrecoverable Also for the record: the mutation-testing results described in the PR body are the author's own report, not independently re-run here — CI below is what this review counts as test evidence. Test evidence — the PR's own CIThe unit suite on the reviewed commit is still running, so the table below is a live snapshot; the finalize job updates it once CI settles. The macOS/Windows/integration rows show Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 Sandboxed verification would settle the one thing CI cannot: 中文说明代码审查在 已核实：
一个如实的保留意见（不阻塞）：PR 描述称被抢救的 checkpoint 留下的更旧证据"已被上一个 checkpoint 的 claim 覆盖"——严格来说，被丢下的记录在游标之后，并不在先前 claim 中；抢救 checkpoint 落地后游标越过它们，它们失去可引用性。这仍严格优于旧行为（整个 Goal 以不可恢复的 另外说明：PR 正文中的变异测试结果系作者自述，本审查未独立重跑——下方 CI 才是本审查计为测试证据的部分。 测试证据——PR 自身的 CI被审提交上的单元测试仍在运行，下表为实时快照；finalize 任务会在 CI 结束后更新它。macOS/Windows/integration 行显示 （CI 表格见上方标记区域，此处不重复。） 沙箱验证可以解决 CI 无法解决的一点： — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 4/5 — a clean, minimal fix for a deadlock I verified mechanically in the base tree; the two reservations below are about framing, not about the code. Stepping back over the whole thing:
The two reservations, neither blocking:
Because CI has not settled on the reviewed commit, approval is deferred rather than posted now: this comment carries the deferred-approval marker, and the finalize job posts the commit-pinned approval once every check on that SHA completes green — withholding it if anything lands red or the head moves. 中文说明置信度：4/5 —— 对一个我在基线代码中从机制上核实过的死锁，给出了干净且最小的修复；下面两点保留意见针对的是表述，不是代码。 整体回顾：
两点保留意见，均不阻塞：
由于被审提交上的 CI 尚未结束，本次不直接批准、改为延迟批准：本评论携带延迟批准标记，finalize 任务会在该 SHA 上所有检查变绿后发布钉住该提交的批准——若有检查变红或 head 移动，则扣下不发。 （标记行见正文，此处不重复其文本。） — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
QwenLM#9835 made a truncated evidence window compact instead of stopping the Goal. That removed the only terminator on the path where compaction runs and gives no relief: a Goal whose evidence rate outruns the catalog pays a checkpoint verifier call every turn, loses evidence every turn, and never converges on its own. A checkpoint is counted as stalled when it comes back holding the maximum number of claims while the window it compacted was already truncated. Compaction has two levers, folding evidence into claims and moving the cursor past what was folded; that combination means the first is exhausted (the next checkpoint can only merge) and the second is not keeping up (eligible evidence was left behind). A busy turn that truncates with room in the claims is not a stall, and a full claim list on a quiet Goal is not either. The streak is persisted as `GoalRecord.checkpointStalls` (absent means zero) so a restart or resume cannot launder it. Any check that finds room resets it, and so do edit and replace. After three consecutive stalls the Goal settles as `usage_limited` with the existing `limitKind: 'evidence_catalog'` and a reason naming what stalled, how many times, and what to do. No new limit kind, no branch in `goalLimitKindForReason`, nothing crosses the wire. Mutation probes (goal-runtime + goal-reducer + goal-checkpoint, 192 tests): no increment -> 3 fail; no reset on an effective checkpoint -> 1; no reset on a quiet check -> 1; threshold >= to > -> 1; parse drops the key -> 1; parse never restores it -> 1; predicate ignores truncation -> 2; predicate ignores the claim cap -> 1; edit stops resetting -> 1. Every other test green in every run.
…wenLM#9975) * feat(goal): stop a Goal whose checkpoints stall three times in a row QwenLM#9835 made a truncated evidence window compact instead of stopping the Goal. That removed the only terminator on the path where compaction runs and gives no relief: a Goal whose evidence rate outruns the catalog pays a checkpoint verifier call every turn, loses evidence every turn, and never converges on its own. A checkpoint is counted as stalled when it comes back holding the maximum number of claims while the window it compacted was already truncated. Compaction has two levers, folding evidence into claims and moving the cursor past what was folded; that combination means the first is exhausted (the next checkpoint can only merge) and the second is not keeping up (eligible evidence was left behind). A busy turn that truncates with room in the claims is not a stall, and a full claim list on a quiet Goal is not either. The streak is persisted as `GoalRecord.checkpointStalls` (absent means zero) so a restart or resume cannot launder it. Any check that finds room resets it, and so do edit and replace. After three consecutive stalls the Goal settles as `usage_limited` with the existing `limitKind: 'evidence_catalog'` and a reason naming what stalled, how many times, and what to do. No new limit kind, no branch in `goalLimitKindForReason`, nothing crosses the wire. Mutation probes (goal-runtime + goal-reducer + goal-checkpoint, 192 tests): no increment -> 3 fail; no reset on an effective checkpoint -> 1; no reset on a quiet check -> 1; threshold >= to > -> 1; parse drops the key -> 1; parse never restores it -> 1; predicate ignores truncation -> 2; predicate ignores the claim cap -> 1; edit stops resetting -> 1. Every other test green in every run. * fix(goal): preserve the stall streak when a check proves no room (QwenLM#9975) finishCheckpointCheck reset checkpointStalls on all three call sites, but only the room branch proved the window had relief. A transient verifier failure or an empty turn now preserves the streak, so intermittent checkpoint-verifier errors cannot launder the count and keep the stall breaker from firing. Also moves withCheckpointStalls out from between takeTurnTokens and its JSDoc. * fix(goal): count unusable checkpoint results toward the stall limit (QwenLM#9975) * fix(goal): surface unusable checkpoint verifier output to the stall breaker (QwenLM#9975) * fix(goal): reset the stall streak when a resume restarts the evidence window QwenLM#9840 landed after this branch opened: an evidence-limited Goal now resumes by repointing the cursor and dropping the checkpoint, which is a different evidence window from the one the streak was counted against. Carrying the count across it spends the new window's allowance on the old window's failures -- a Goal resumed at two stalls would stop again after a single stalled checkpoint. A resume that does NOT restart the window (paused, blocked) keeps the streak: that Goal comes back to the same window, so what it learned about that window is still true. Mutation probe: removing the reset fails exactly the new resume test (75 others green). --------- Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com>
|
Released in v0.22.2. |
3 participants
What this PR does
Evidence catalog previews are now capped to 240 UTF-8 bytes on a code point boundary, rather than to 240 characters, at the two points a catalog entry is built. And a truncated evidence window now compresses what it captured instead of stopping the Goal — only a window that captured nothing at all still fails.
Why it's needed
The catalog budget is 24,000 bytes. The preview limit that feeds it was 240 characters. In UTF-8 those units differ by up to four times, so the guard held only for ASCII.
The consequence is not a tuning problem, it is a Goal that cannot run in some languages. A checkpoint may carry 32 claims; each is projected into the catalog as a preview plus a small envelope. With ASCII that is about 430 bytes per entry, so a full checkpoint occupies 13,760 bytes — 57% of the budget, leaving room for roughly 23 more records. With Chinese the same legal checkpoint serializes to about 29,000 bytes, over the cap on its own, before a single new record has been scanned. The window is then marked truncated, and
shouldCheckpointrequired!truncated— so the one state compaction exists to resolve was the one state it refused to run in. The runtime read that as unrecoverable and stopped the Goal asusage_limited, whichreduceGoalResumerefuses to resume, and there was nothing left to salvage because no candidate evidence had been collected yet.An English Goal never reached that state. A Chinese one could not avoid it: the failure arrives on the first checkpoint that fills its claim budget, regardless of how the run is going.
The second half follows from the first. A window that overflows its budget is precisely the state compaction is for — the newest evidence that did fit is exactly what a checkpoint folds into claims, and the older evidence left behind is already covered by the previous checkpoint's claims. Refusing to compress there and stopping the Goal instead is strictly worse than compressing what is available.
Reviewer Test Plan
How to verify
Run a Goal whose objective and work are in a non-Latin script — Chinese is the reported case — long enough for it to take a checkpoint whose claims fill the claim budget. Before this change the next window is truncated on arrival and the Goal stops as
usage_limitedwithlimitKind: 'evidence_catalog', and/goal resumeis refused. After it, the checkpoint stays comfortably inside the catalog budget and the run continues. Then force an overflow deliberately (more than 100 post-cursor records) and confirm the Goal now takes a checkpoint and stays active rather than stopping.Two tests carry the change.
does not start truncated under a full checkpoint of multi-byte claimsbuilds a goal carrying 32 claims of 2,000 Chinese characters — the maximum a checkpoint may hold — and asserts the window is not truncated.compresses a truncated window instead of stopping the Goaldrives 101 post-cursor records through the runtime and asserts the checkpoint verifier is called, the Goal stays active, and the journal records acheckpointrather than ausage_limitedcause.Both were mutation-checked rather than merely observed to pass. Reverting the byte cap on checkpoint claims to the old character slice fails the multi-byte test and leaves the other 29 in that file green. Restoring the
!truncatedgate onshouldCheckpointfails the salvage test and leaves the other 108 in the runtime file green. Both mutations were reverted and the suites re-run clean.The existing
moves to usage_limited when checkpoint %s failscase was split:flushandreadstill assertusage_limited, because a transient I/O failure is unrelated to this change, whiletruncatedbecame the salvage test above. One existing multi-byte fixture was rescaled from 26 records to 55 — its intent is that a window crossing the 19,200-byte threshold arms a checkpoint, and a correctly capped CJK entry now costs about 364 bytes instead of about 910, so the same record count no longer reaches the threshold. The assertion is unchanged; only the fixture size moved.npx vitest run packages/core/src/goals/passes 388 tests across 15 files.npx tsc --noEmitinpackages/coreis clean apart from pre-existing@types/nodeskew insrc/utils/tool-result-boundary-diagnostics.ts, a file this PR does not touch.prettierandeslintare clean on all four changed files.Evidence (Before & After)
Before, from the reported session: a Chinese Goal ran 27 turns and stopped as
usage_limitedwithThe current Goal revision exceeded the bounded evidence catalog. Automatic retries cannot recover. Edit or replace the Goal before resuming it.— 27 of 60 files written, no output, and/goal resumerefused.After: a full 32-claim Chinese checkpoint occupies roughly 13,000 bytes rather than 29,000, so the window is not truncated on arrival; and when a window does overflow, the runtime compresses it and the Goal stays active.
Tested on
Environment (optional)
Linux, Node.js 22, unit tests only.
Risk & Scope
evidenceContent, which is capped separately and unchanged here. ASCII previews are byte-for-byte unchanged, since the two units already agreed there.CATALOG_BYTE_LIMIT,CHECKPOINT_BYTE_THRESHOLDor the claim budget — the units are reconciled at the values already in the tree rather than retuned. It does not add compaction-effectiveness detection: a checkpoint that compresses but does not shrink enough will still overflow again, which is a separate change. Windows and macOS were not exercised locally and remain covered by CI.packages/core/src/goals, with no cross-package change.Linked Issues
None — the failure is reported here rather than in an existing issue.
中文说明
本 PR 做了什么
证据目录的 preview 现在按 240 个 UTF-8 字节、在码点边界上截断，而不是按 240 个字符；截断施加在构造 catalog entry 的两个汇聚点上。另外，被截断的证据窗口现在会压缩它已经捕获的内容，而不是停掉 Goal —— 只有「什么都没捕获到」的窗口才仍然失败。
为什么需要
catalog 预算是 24,000 字节，而喂给它的 preview 上限是 240 个字符。UTF-8 下这两个单位最多相差四倍，因此这道防线只在 ASCII 下成立。
后果不是调参问题，而是 Goal 在某些语言下根本跑不了。一个 checkpoint 最多可以携带 32 条 claim，每条会被投影成 catalog 中的一条 preview 加一个小信封。ASCII 下每条约 430 字节，满载 checkpoint 占 13,760 字节 —— 预算的 57%，之后还能装大约 23 条记录。中文下同样合法的 checkpoint 序列化后约 29,000 字节，在还没扫描任何一条新记录之前，光它自己就越过了上限。窗口于是被标记为 truncated，而
shouldCheckpoint要求!truncated—— 压缩本该解决的那个状态，恰恰是它拒绝运行的那个状态。运行时把这读作不可恢复，将 Goal 停为usage_limited（reduceGoalResume拒绝恢复的那个状态），而且此时没有任何可抢救的东西，因为一条候选证据都还没被收集。英文 Goal 永远到不了这个状态；中文 Goal 则无法避免它：故障在第一个填满 claim 预算的 checkpoint 上就会到来，与运行得好不好无关。
第二半是第一半的推论。一个越过预算的窗口，正是压缩存在的意义 —— 装得下的那些最新证据，恰恰是 checkpoint 要折叠成 claim 的东西，而被留下的更旧证据早已被上一个 checkpoint 的 claim 覆盖。在那里拒绝压缩、转而停掉 Goal，严格劣于压缩手头能压的部分。
评审者测试计划
如何验证
用非拉丁文字（报告的案例是中文）的 objective 跑一个 Goal，跑到它做出一个 claim 填满预算的 checkpoint。改动之前，下一个窗口在构造时就是 truncated，Goal 以
usage_limited与limitKind: 'evidence_catalog'停止，/goal resume被拒绝。改动之后，checkpoint 稳稳落在 catalog 预算之内，运行继续。然后刻意制造一次溢出（游标之后超过 100 条记录），确认 Goal 现在会做一次 checkpoint 并保持 active，而不是停止。两个测试承载这次改动。
does not start truncated under a full checkpoint of multi-byte claims构造一个携带 32 条、每条 2,000 个中文字符 claim 的 goal（checkpoint 可容纳的上限），断言窗口未被截断。compresses a truncated window instead of stopping the Goal把 101 条游标后记录送进运行时，断言 checkpoint verifier 被调用、Goal 保持 active、journal 记录的是checkpoint而不是usage_limited。两者都做了变异检验而不只是「跑通了」。把 checkpoint claim 的字节封顶改回旧的字符切片，恰好让多字节那个测试失败，同文件其余 29 个仍绿。把
shouldCheckpoint上的!truncated门恢复，恰好让抢救那个测试失败，runtime 文件其余 108 个仍绿。两次变异都已还原并重跑干净。既有的
moves to usage_limited when checkpoint %s fails用例被拆开：flush与read仍断言usage_limited，因为瞬时 I/O 失败与本次改动无关；truncated则变成了上面那个抢救测试。另有一个既有的多字节 fixture 从 26 条记录扩到 55 条 —— 它的意图是「跨过 19,200 字节阈值的窗口会武装 checkpoint」，而正确封顶后的 CJK entry 现在约 364 字节而非约 910，同样的记录数不再够到阈值。断言未变，只是 fixture 规模改了。npx vitest run packages/core/src/goals/通过 15 个文件共 388 个测试。packages/core的npx tsc --noEmit干净，仅剩src/utils/tool-result-boundary-diagnostics.ts中预先存在的@types/node偏斜，本 PR 未触及该文件。四个改动文件的prettier与eslint均干净。证据（修复前后）
修复前，来自已报告的 session：一个中文 Goal 跑了 27 轮后以
usage_limited停止，原因是The current Goal revision exceeded the bounded evidence catalog. Automatic retries cannot recover. Edit or replace the Goal before resuming it.—— 60 个文件写了 27 个、没有产出，且/goal resume被拒绝。修复后：满载 32 条 claim 的中文 checkpoint 约占 13,000 字节而非 29,000，窗口在构造时不再是 truncated；而当窗口确实溢出时，运行时会压缩它，Goal 保持 active。
测试平台
环境（可选）
Linux、Node.js 22，仅单元测试。
风险与范围
evidenceContent读取完整内容，那一路有独立的封顶，本 PR 未改动。ASCII 的 preview 逐字节不变，因为那里两个单位本来就一致。CATALOG_BYTE_LIMIT、CHECKPOINT_BYTE_THRESHOLD或 claim 预算 —— 是在树上已有的数值下把单位对齐，而不是重新调参。它也不加入「压缩是否有效」的检测：一个压缩了但没缩够的 checkpoint 仍会再次溢出，那是另一次改动。Windows 与 macOS 未在本地验证，仍由 CI 覆盖。packages/core/src/goals下两个文件，新增 49 行、删除 7 行生产代码，无跨包改动。关联 Issue
无 —— 该故障在此处首次记录，没有对应的既有 issue。