feat(core): add bounded Goal evidence verification - #7639
Conversation
|
Thanks for the PR! Template looks good ✓ Problem: This is a planned feature slice — the second focused PR split from #7494, following #7517 which established the Goal v3 protocol. The "problem" is a design requirement: Goals must not self-certify completion from ungrounded model assertions. Not a bug fix; a deliberate architectural addition to the Goal v3 contract. Direction: Aligned. The existing Size: Core paths touched ( Approach: The scope feels right for the stated goal. Two cohesive modules (evidence catalog + verifier) with clear boundaries. Every edit serves the stated purpose — no drive-by refactors or unrelated changes. The deferred scope (transcript recording, runtime scheduling, CLI/TUI integration) is explicitly listed and reasonable for a focused slice. Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ 问题:这是一个计划中的功能切片——从 #7494 拆出的第二个聚焦 PR,接续 #7517 建立的 Goal v3 协议。"问题"是设计需求:Goal 不能仅凭模型自述标记完成。不是 bug 修复,而是 Goal v3 契约的架构性补充。 方向:对齐。现有 规模:触及核心路径( 方案:范围合理。两个内聚模块(证据目录 + 验证器),边界清晰。所有改动服务于既定目标,无顺手重构或无关变更。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code ReviewIndependent proposal: For "bounded evidence verification for Goals", I would: (1) classify transcript records by provenance (user/assistant/tool), (2) build a newest-first bounded catalog with entry and byte limits, (3) validate terminal proposal references against the catalog, (4) enforce blocker-specific coverage rules, (5) create an independent verifier via a tool-free side query with strict schema, timeout, and request size cap. Comparison: The PR matches this proposal closely. Two well-separated modules — Findings: No critical blockers. The implementation is solid:
One minor observation (non-blocking): Conventions: ESM ✓, no CI Test EvidenceAll checks on commit
No failures. The PR author reports 9 test files, 176 tests passing locally (author's claim — not independently verified here since we don't run PR code). Not verified: real-scenario TUI testing — this PR defines a non-UI contract (no CLI/TUI surface), so tmux testing is not applicable. A maintainer can trigger the isolated 中文说明代码审查独立方案: 对"Goal 有界证据验证",我会:(1) 按来源分类会话记录,(2) 构建最新优先的有界目录,(3) 验证终态提案引用,(4) 强制阻塞类型覆盖规则,(5) 通过无工具独立查询创建验证器。 对比: PR 与此方案高度一致。两个分离良好的模块,无遗漏的更简路径。 发现: 无关键阻断。 实现扎实:各层级均有边界限制;来源一致性检查正确交叉验证 type/subtype/provenance;验证器系统提示明确要求将证据内容视为不可信数据; 一个非阻断观察: CI 测试证据所有检查通过(ubuntu 测试 ✅,web-shell E2E ✅,其余 skipped)。无失败。本 PR 定义非 UI 契约,tmux 测试不适用。 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 3/5 — clean review, but the Stage 0 core-path escalation (876 production lines) needs a maintainer's sign-off. This is a well-executed feature slice. The evidence catalog and verifier are cohesive, the bounds are enforced at every layer, the error taxonomy is machine-readable, and the tests are thorough (176 tests covering every error code path, boundary condition, and the verifier's cancellation/timeout behavior). The code reads like someone who thought carefully about what could go wrong — duplicate UUIDs, turn re-entry, stale revisions, forged provenance, oversized payloads — and handled each case explicitly. My independent proposal matched the PR's approach; I didn't find a materially simpler path. The scope is tight for what it delivers: two modules, one barrel export, no drive-by changes. The deferred integration work (runtime, CLI, TUI) is correctly scoped out. The reason this isn't a 5/5 is purely policy: 876 production lines in ⏸️ Deferring to @wenshao — the review is clean and CI is green, but this adds 876 production lines to core ( 中文说明置信度:3/5 — 审查干净,但 Stage 0 核心路径升级(876 行生产代码)需维护者签字。 这是一个执行良好的功能切片。证据目录和验证器内聚,各层边界强制执行,错误分类机器可读,测试全面(176 项测试覆盖所有错误路径、边界条件和验证器取消/超时行为)。 我的独立方案与 PR 方案一致,未找到更简路径。范围紧凑:两个模块、一个桶导出、无顺手改动。 不是 5/5 的原因纯粹是策略:876 行核心生产代码超过 500 行维护者关注阈值。代码质量无疑——升级是关于架构所有权,而非对正确性的怀疑。 ⏸️ 转交 @wenshao — 审查干净、CI 绿色,但本 PR 向核心( — Qwen Code · qwen3.8-max-preview Reviewed at |
|
⏸️ Deferring to @wenshao — this |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
| if ( | ||
| proposal.blockerKind === 'authority' || | ||
| proposal.blockerKind === 'external' | ||
| ) { |
There was a problem hiding this comment.
[Suggestion] validateBlockerCoverage silently treats undefined blockerKind as 'repeated' — Failure scenario: a GoalTerminalProposal with { status: 'blocked', blockerKind: undefined } is valid per the type (blockerKind?: 'authority' | 'external' | 'repeated'), but falls through both explicit checks here into the "repeated blocker" 3-turn-coverage path. If the coverage check fails, the error says "A repeated blocker requires evidence from the current and two immediately preceding Goal turns" — misleading because blockerKind was undefined, not 'repeated'.
| if ( | |
| proposal.blockerKind === 'authority' || | |
| proposal.blockerKind === 'external' | |
| ) { | |
| if ( | |
| proposal.blockerKind === 'authority' || | |
| proposal.blockerKind === 'external' | |
| ) { |
Consider adding an explicit guard for blockerKind === undefined before this branch, or widening the error message to cover the unspecified-kind case.
— qwen3.7-max via Qwen Code /review
| if (context.goalId !== input.goal.goalId) { | ||
| throw new InvalidGoalEvidenceReferenceError( | ||
| 'wrong_goal_id', |
There was a problem hiding this comment.
[Suggestion] The wrong_goal_id and wrong_revision error branches are reachable but have no test coverage — Failure scenario: a record with valid provenance and a parseable goalContext whose goalId or revision differs from the input goal reaches these branches and throws the specific error codes. catalogEvidence silently excludes such records, so the catalog lookup at reference_not_catalogued would never fire for them — but no test confirms the correct, specific error is thrown. If a future change reorders these checks or broadens catalog inclusion, the wrong error code (or a silent pass) would go undetected.
Add test cases that construct records with a mismatched goalId and a mismatched revision (both with valid provenance and parseable context) and assert { code: 'wrong_goal_id' } and { code: 'wrong_revision' } respectively.
— qwen3.7-max via Qwen Code /review
Code Review — #7639
|
|
@qwen-code /triage |
|
Triage re-run completed without a new review. The stage comments above were updated with the latest result. View workflow run. |
✅ Local build & test verification — #7639
|
| # | Guarantee broken | Result |
|---|---|---|
| M1 | Verifier payload whitelists named fields (no full-record / preview leak) |
✗ 1 failed |
| M2 | Hidden reasoning (thought) excluded from cited content |
✗ 1 failed |
| M3 | Catalog never expands records older than the bounded window (lazy getter) | ✗ 1 failed |
| M4 | 64 KB request cap enforced before the provider call | ✗ 1 failed |
| M5 | Strict parse rejects extra keys | ✗ 1 failed · 3 sibling cases stay green |
| M6 | Provenance forgery guard (a user record cannot claim assistant_output) |
✗ 1 failed |
| M7 | Immediate blocker requires cited user/tool evidence | ✗ 2 failed (authority + external) |
M5 is fully discriminating (only the extra:true it.each case flips), and M7 breaks both the authority and external blocker cases — confirming these aren't blanket assertions.
3 · Independent confirmation of review Note #1 (byte-framing)
I reproduced the one substantive note from the static review. The catalog sums per-entry Buffer.byteLength(JSON.stringify(entry)), but the serialized array adds [, ], and (n−1) commas. At 100 retained entries the serialized catalog reaches 24 001 B — 1 byte over the 24 000 cap (see the probe table in the second image). Real but ≤ ~0.4 % and determinism-safe → non-blocking, exactly as flagged.
Verdict
Verified — merge-ready. Executable evidence now backs the earlier clean static review: build + typecheck pass, 177/177 goals tests pass, and each security guarantee of the two new modules — whitelist request construction, hidden-reasoning exclusion, provenance forgery guard, bounded catalog / reference / verifier budgets, pre-provider size cap, cancellation + timeout — is enforced by a load-bearing test. Runtime / TUI / WebShell E2E remains correctly deferred to the follow-up slices; Windows / Linux not exercised locally.
中文说明
✅ 本地构建与测试验证 — #7639 feat(core): add bounded Goal evidence verification
我在 PR HEAD 2c24ca5a8 上、用一个隔离的干净 worktree 真实构建并运行了测试套件 —— 这正是之前静态审查明确未做的部分(“未在本地跑测试套件——该 checkout 早于 v3 基线”)。PR 中的每一项声明都能复现,并且我额外用对抗性变异(mutation)证明了这些测试是有实际约束力的。
环境: macOS(darwin 24.6)· Node v22.23.1 · packages/core 从源码构建 · PR HEAD 的隔离检出(依赖 goal-protocol.ts / sideQuery.ts / baseLlmClient.ts 在该分支上均可解析)。
1 · 构建、类型检查与完整套件 —— 全绿
npm run build(tsc)✓ ·npm run typecheck(tsc --noEmit)✓- 整个
goals/包:9 个文件 / 177 个测试通过 —— 复现 PR 的“9 个文件”声明。(正文写的是 176;这 1 个测试的差异位于相邻的 goals 测试文件;两个新增文件恰好贡献 24 个测试,且是确定性的。) - 仅两个 PR 文件(
goal-evidence.test.ts+goal-verifier.test.ts):24 / 24 通过。
(见上方第一张截图。)
2 · 对抗性证据 —— 测试有实际约束力,而非空测
绿色套件只有在“破坏某个保证时对应测试会变红”时才有意义。我做了区分性的 A/B:在实现中破坏某一项保证,只运行声称守护该保证的测试,确认它变红,再恢复 → 回到 24 全绿。全部 7 个变异都精确命中各自的目标测试。
| # | 被破坏的保证 | 结果 |
|---|---|---|
| M1 | verifier 载荷仅白名单命名字段(不泄露整条记录 / preview) |
✗ 1 失败 |
| M2 | 隐藏推理(thought)不进入被引用内容 |
✗ 1 失败 |
| M3 | 目录绝不展开窗口之外的更旧记录(惰性 getter 计数) | ✗ 1 失败 |
| M4 | 64 KB 请求上限在调用 provider 之前强制 | ✗ 1 失败 |
| M5 | 严格解析拒绝多余字段 | ✗ 1 失败 · 另外 3 个同族用例保持绿色 |
| M6 | provenance 伪造防护(user 记录不能冒充 assistant_output) |
✗ 1 失败 |
| M7 | 即时阻塞必须引用用户/工具证据 | ✗ 2 失败(authority + external) |
M5 完全可区分(仅 extra:true 那个 it.each 用例翻红),M7 同时破坏 authority 与 external 两个用例 —— 说明它们不是笼统断言。
3 · 独立复现审查 Note #1(字节-框架计数)
我复现了静态审查中唯一实质性的一条。目录按每条 Buffer.byteLength(JSON.stringify(entry)) 求和,但序列化后的数组还会加上 [、] 和 (n−1) 个逗号。在保留 100 条时,序列化后的目录达到 24 001 B —— 超出 24 000 上限 1 字节(见第二张图中的探针表)。真实存在但 ≤ 约 0.4%,且不破坏确定性 → 非阻塞,与该 note 描述完全一致。
结论
已验证 —— 可合并。 现在有可执行证据支撑此前干净的静态审查:构建 + 类型检查通过,177/177 个 goals 测试通过,两个新模块的每一项安全保证 —— 白名单构造请求、隐藏推理剔除、provenance 伪造防护、有界目录/引用/verifier 预算、调用 provider 前的大小上限、取消 + 超时 —— 都由一个有实际约束力的测试守护。Runtime / TUI / WebShell 的端到端验证按设计正确地留给后续切片;Windows / Linux 未在本地验证。
Verified locally by the maintainer at PR head 2c24ca5a8 · isolated worktree · macOS · Node 22.23.1. Mutation harness & byte-framing probe are reproducible scripts.


What this PR does
Adds the bounded evidence and independent verification layer for Goal v3 as the second focused slice split from #7494.
Goal-owned transcript records are classified by provenance and exact Goal identity, revision, and turn lineage. The evidence catalog exposes bounded previews of the newest eligible records, while terminal proposals must cite catalogued UUIDs and satisfy reference-count, duplicate-reference, and total-content byte limits. Completion and blocker proposals are then judged by a deterministic, tool-free side query with a strict response schema, cancellation support, timeout handling, and a total request-size limit.
Why it's needed
A Goal must not mark itself complete from an ungrounded model assertion. Evidence needs to prove the right kind of fact: user input proves authority, delivered assistant output proves only what was sent, and tool results may prove external state. References from another Goal, revision, transcript prefix, or broken turn lineage must fail before verification.
The original combined PR also allowed evidence work to grow with session length. This slice defines explicit catalog, lineage, reference, cited-content, and verifier-request budgets. Catalog truncation keeps the newest evidence and stops expanding older records once the budget is reached, so the current turn remains usable in long-running Goals.
Reviewer Test Plan
How to verify
Confirm that only real user input, delivered assistant output, and tool results with matching Goal ownership enter the evidence catalog. Confirm that internal runtime prompts, system records, mismatched provenance, pre-cursor records, wrong Goal identities, stale revisions, malformed contexts, turn re-entry, and a non-tail permit are rejected.
Confirm that the catalog retains the newest evidence, limits previews to 240 characters, limits the serialized catalog to 100 entries and 24 KB, and does not expand records older than its retained window. Confirm that terminal proposals require 1–12 unique catalogued references with at most 24 KB of cited content.
Confirm that immediate authority/external blockers require user or tool evidence and repeated technical blockers require evidence from the latest three turns. Confirm that the independent verifier receives only bounded protocol fields, uses no tools, accepts only exact
accept/rejectJSON, honors cancellation and timeout, and rejects requests above 64 KB before calling the provider.Local result: 9 test files passed, 176 tests passed. Core type checking and build passed.
Evidence (Before & After)
N/A — this PR defines a non-UI evidence and verification contract. Runtime, TUI, and WebShell E2E evidence will be attached to their focused follow-up PRs.
Tested on
Environment (optional)
Node.js 24, no sandbox.
Risk & Scope
Linked Issues
Follows #7517. Split from #7494.
中文说明
这个 PR 做了什么
作为从 #7494 拆出的第二个聚焦改动,为 Goal v3 增加有界证据和独立验证层。
Goal 所属的会话记录会按照来源以及精确的 Goal 身份、修订号和轮次谱系分类。证据目录只暴露最新合格记录的有界预览;终态提案必须引用目录中的 UUID,并满足引用数量、重复引用和内容总字节限制。随后使用确定性、无工具的独立查询判断完成或阻塞提案,同时强制严格响应结构、取消、超时和请求总大小限制。
为什么需要
Goal 不能仅凭模型自述将自己标记为完成。不同证据只能证明对应事实:用户输入证明权限或选择,助手输出只能证明内容已发送,工具结果才可以证明外部状态。来自其他 Goal、其他修订、游标之前或损坏轮次谱系的引用必须在验证前失败。
原始合并 PR 中的证据处理还可能随会话长度持续增长。本切片明确规定目录、谱系、引用、引用内容和 verifier 请求预算。目录截断会保留最新证据,并在达到预算后停止展开更旧记录,从而保证长时间运行的 Goal 仍能使用当前轮证据。
Reviewer 测试计划
如何验证
确认只有具备匹配 Goal 所有权的真实用户输入、已交付助手输出和工具结果能够进入证据目录。确认内部运行时提示、系统记录、来源不匹配、游标前记录、错误 Goal 身份、过期修订、畸形上下文、轮次重入和非尾部 permit 都会被拒绝。
确认目录保留最新证据,预览最多 240 个字符,序列化目录最多 100 项和 24 KB,并且不会展开保留窗口之外的更旧记录。确认终态提案必须提供 1–12 个唯一且在目录中的引用,引用内容总量最多 24 KB。
确认即时权限或外部阻塞必须引用用户或工具证据,重复技术阻塞必须覆盖最近三个轮次。确认独立 verifier 只接收有界协议字段、不使用工具、只接受精确的
accept/rejectJSON、支持取消和超时,并在调用模型前拒绝超过 64 KB 的请求。本地结果:9 个测试文件通过,共 176 项测试通过。Core 类型检查和构建通过。
证据(改动前后)
不适用——本 PR 定义非 UI 的证据和验证契约。Runtime、TUI 和 WebShell 的端到端证据会附在对应的后续 PR 中。
测试平台
环境(可选)
Node.js 24,无沙箱。
风险与范围
关联事项
接续 #7517,从 #7494 拆分。