Skip to content

test(channels): add memory recall evaluation baseline - #7220

Merged
wenshao merged 4 commits into
QwenLM:mainfrom
qwen-code-dev-bot:codex/issue-7216-channel-memory-recall-eval
Jul 20, 2026
Merged

test(channels): add memory recall evaluation baseline#7220
wenshao merged 4 commits into
QwenLM:mainfrom
qwen-code-dev-bot:codex/issue-7216-channel-memory-recall-eval

Conversation

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

What this PR does

This PR adds a deterministic, synthetic multilingual evaluation baseline for channel memory recall. It covers English, Chinese, Japanese, Korean, mixed-script, fallback, no-result, ranking, and prompt-budget behavior, and freezes quality thresholds for Recall@3, top-1 accuracy, no-result precision, selected-entry count, and selected text size.

The fixture contract is strict: case inventory, category counts, label references, exact ordering for critical cases, metric denominators, and fixed budget limits are validated so future fixture drift cannot silently weaken the baseline. The evaluation calls the existing production selector directly and does not change runtime behavior.

Why it's needed

Channel memory recall now has caching and multilingual lexical selection, but it lacked a reproducible quality floor. This baseline makes later telemetry and semantic-recall work measurable while protecting current multilingual, no-result, fallback, ranking, and budget behavior from regression.

Reviewer Test Plan

How to verify

Run the channel-base recall evaluation and confirm all 36 labeled cases load, the fixed denominators remain 31 positive cases and 5 no-result cases, Recall@3 is at least 0.90, top-1 accuracy is at least 0.85, no-result precision is exactly 1.00, and every result stays within 3 entries and 1,200 code points. Run the existing ChannelBase and CLI channel tests and confirm foreground and daemon memory wiring, revision caching, mutation invalidation, target isolation, access gates, and read-failure handling remain unchanged.

Evidence (Before & After)

N/A. This is a test-only change with no user-visible or runtime behavior changes.

Tested on

OS Status
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

Environment (optional)

Node.js 22 workspace dependencies installed with npm install.

Risk & Scope

  • Main risk or tradeoff: The checked-in fixture intentionally measures a bounded synthetic quality floor rather than claiming broad natural-language coverage.
  • Not validated / out of scope: Production telemetry, semantic retrieval, model reranking, embeddings, storage changes, access-control changes, and live channel credentials.
  • Breaking changes / migration notes: None.

Linked Issues

Fixes #7216

中文说明

本 PR 做了什么

本 PR 为 channel memory recall 增加了一套确定性、完全合成的多语言评估基线,覆盖英文、中文、日文、韩文、混合脚本、fallback、无结果、排序和 prompt 预算行为,并固定 Recall@3、Top-1 准确率、无结果精确率、选中条目数量和选中文本大小的质量阈值。

夹具契约采用严格校验:固定用例清单和分类数量,检查标签引用、关键用例的精确顺序、指标分母和固定预算上限,避免未来修改夹具时悄悄削弱基线。评估直接调用现有生产 selector,不改变任何运行时行为。

为什么需要

Channel memory recall 已具备缓存和多语言词法选择,但缺少可复现的质量底线。这套基线让后续 telemetry 和语义召回工作可以被量化,同时保护现有多语言、无结果、fallback、排序和预算行为不发生回归。

Reviewer 测试计划

如何验证

运行 channel-base recall 评估,确认 36 个标注用例全部加载,固定分母仍为 31 个正向用例和 5 个无结果用例,Recall@3 不低于 0.90,Top-1 准确率不低于 0.85,无结果精确率严格等于 1.00,并且每个结果不超过 3 个条目和 1,200 个 Unicode code point。运行现有 ChannelBase 和 CLI channel 测试,确认前台及 daemon memory wiring、revision cache、mutation invalidation、target isolation、access gate 和读取失败处理均保持不变。

前后证据

N/A。本次仅增加测试,不包含用户可见或运行时行为变化。

测试环境

OS 状态
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

Node.js 22,工作区依赖通过 npm install 安装。

风险与范围

  • 主要风险或取舍:提交的夹具用于衡量边界明确的合成质量底线,不代表覆盖所有自然语言表达。
  • 未验证或范围外:生产 telemetry、语义检索、模型重排、embedding、存储变化、访问控制变化和真实 channel 凭据。
  • 破坏性变化或迁移说明:无。

关联 Issue

Fixes #7216

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: this is a deliberate testing gap, not a theoretical concern. Issue #7216 explicitly requests a reproducible quality floor for the existing channel memory recall selector, which already ships in production with caching and multilingual lexical selection but has no evaluation baseline. The issue is well-specified with acceptance criteria and labeled status/ready-for-agent.

Direction: aligned — adding deterministic test coverage for an existing production code path. No product direction concerns.

Size: not applicable (files are in packages/channels/base/src/, not a core module path). 1,074 additions across 2 new files, all test/fixture code with zero production changes.

Approach: the scope feels right. Two files — a 36-case synthetic fixture and a test that calls the production selectRelevantChannelMemory directly, computes Recall@3, top-1 accuracy, no-result precision, and budget compliance, then asserts frozen thresholds. The fixture contract validation (strict schema, case inventory, category counts, label references) is thorough but justified by the issue's explicit requirement that future fixture drift cannot silently weaken the baseline. No unrelated changes or scope creep.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题:这是一个明确的测试缺口,而非理论性问题。Issue #7216 明确要求为现有的 channel memory recall 选择器建立可复现的质量底线——该选择器已在生产环境中运行,支持缓存和多语言词法选择,但缺少评估基线。Issue 规格完整,带有验收标准,并标记为 status/ready-for-agent

方向:对齐——为已有生产代码路径添加确定性测试覆盖,无产品方向顾虑。

规模:不适用(文件位于 packages/channels/base/src/,非核心模块路径)。共 1,074 行新增,分布在 2 个新文件中,全部为测试/夹具代码,零生产代码变更。

方案:范围合理。两个文件——36 个合成用例的夹具和一个直接调用生产 selectRelevantChannelMemory 的测试,计算 Recall@3、Top-1 准确率、无结果精确率和预算合规性,然后断言固定阈值。夹具契约校验(严格 schema、用例清单、分类数量、标签引用)较为详尽,但这是 issue 明确要求的——确保未来修改夹具时不会悄悄削弱基线。无无关改动或范围蔓延。

进入代码审查 🔍

Qwen Code · qwen3.7-max

Reviewed at 0c19d1404f551537ba22bd65e87e41b0a8b6df47 · re-run with @qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Code Review

Independent proposal: given the issue's acceptance criteria, I would create a JSON fixture with ~36 labeled cases across en/zh/ja/ko/mixed/fallback/no-result/ranking/budget categories, write a test that loads and validates the fixture schema, calls the production selectRelevantChannelMemory for each case, computes standard IR metrics (Recall@3, top-1 accuracy, no-result precision), and asserts frozen thresholds plus determinism. Two files, no production changes.

Comparison with the diff: the PR matches this approach almost exactly. The implementation is clean and well-structured:

  • The fixture validation is thorough — assertKnownFields rejects unknown fields, parseStringArray enforces uniqueness, assertReferencesBelongToEntries validates cross-references. This is the "strict fixture contract" the issue asks for, and it prevents silent fixture drift.
  • The evaluate function correctly computes Recall@3 (per-case recall averaged), top-1 accuracy, and no-result precision (precision of empty-selection predictions). The budget check validates both the production constants and the actual results.
  • The requiredSelectedIds map hardcodes exact ordered selections for 12 critical cases — fallback ordering, tie-breaking, budget skipping, and all no-result cases. This is the strongest form of regression protection.
  • The first test (counts empty selections on positive cases against no-result precision) is a unit test for the evaluation logic itself, not the fixture — a good sanity check that false negatives count against precision.

No critical blockers. No AGENTS.md violations. The code follows project conventions (ESM .js imports, vitest, collocated tests, no any types).

Testing

Ran on the PR branch (pr-7220) in tmux:

$ cd packages/channels/base && npx vitest run src/channel-memory-recall-eval.test.ts

 RUN  v3.2.4 /home/github-runner/actions-runner-2/_work/qwen-code/qwen-code/packages/channels/base

 ✓ src/channel-memory-recall-eval.test.ts (6 tests) 22ms

 Test Files  1 passed (1)
      Tests  6 passed (6)
   Start at  17:47:55
   Duration  453ms (transform 89ms, setup 0ms, collect 80ms, tests 22ms, environment 0ms, prepare 125ms)

Existing recall tests (no regression):

$ npx vitest run src/channel-memory-recall.test.ts

 RUN  v3.2.4 /home/github-runner/actions-runner-2/_work/qwen-code/qwen-code/packages/channels/base

 ✓ src/channel-memory-recall.test.ts (17 tests) 9ms

 Test Files  1 passed (1)
      Tests  17 passed (17)
   Start at  17:48:36
   Duration  257ms (transform 55ms, setup 0ms, collect 40ms, tests 9ms, environment 0ms, prepare 80ms)

ChannelBase tests (496 tests, no regression):

 ✓ src/ChannelBase.test.ts (496 tests) 8210ms

 Test Files  1 passed (1)
      Tests  496 passed (496)
   Duration  8.97s

Typecheck and build both pass cleanly.

中文说明

代码审查

独立方案: 根据 issue 的验收标准,我会创建一个包含约 36 个标注用例的 JSON 夹具,覆盖 en/zh/ja/ko/mixed/fallback/no-result/ranking/budget 类别,编写测试加载并校验夹具 schema,对每个用例调用生产 selectRelevantChannelMemory,计算标准 IR 指标(Recall@3、Top-1 准确率、无结果精确率),并断言固定阈值和确定性。两个文件,零生产代码变更。

与 diff 的对比: PR 几乎完全匹配此方案。实现干净、结构良好:

  • 夹具校验非常详尽——assertKnownFields 拒绝未知字段,parseStringArray 强制唯一性,assertReferencesBelongToEntries 校验交叉引用。这正是 issue 要求的"严格夹具契约",防止夹具悄悄漂移。
  • evaluate 函数正确计算 Recall@3(按用例平均)、Top-1 准确率和无结果精确率(空选择预测的精确率)。预算检查同时校验生产常量和实际结果。
  • requiredSelectedIds 映射为 12 个关键用例硬编码了精确有序选择——fallback 排序、平局打破、预算跳过和所有无结果用例。这是最强的回归保护形式。
  • 第一个测试(counts empty selections on positive cases against no-result precision)是评估逻辑本身的单元测试,而非夹具的——很好地验证了假阴性会计入精确率。

无关键阻塞项。无 AGENTS.md 违规。代码遵循项目约定(ESM .js 导入、vitest、共置测试、无 any 类型)。

测试

在 PR 分支(pr-7220)的 tmux 中运行:

  • 新评估测试:6 个测试全部通过
  • 现有 recall 测试:17 个测试全部通过(无回归)
  • ChannelBase 测试:496 个测试全部通过(无回归)
  • Typecheck 和 build 均干净通过

Qwen Code · qwen3.7-max

Reviewed at 0c19d1404f551537ba22bd65e87e41b0a8b6df47 · re-run with @qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Confidence: 5/5 — clean test-only PR that does exactly what the issue asks for, with zero production risk.

This is a straightforward, well-scoped evaluation baseline. Two new files, no production changes, and the test directly exercises the existing selectRelevantChannelMemory selector against 36 synthetic multilingual cases. The fixture contract is strict enough to catch drift, the metrics are standard (Recall@3, top-1 accuracy, no-result precision), and the hardcoded requiredSelectedIds for 12 critical cases gives the strongest form of regression protection. All tests pass — the 6 new eval tests, the 17 existing recall tests, and the 496 ChannelBase tests. Typecheck and build are clean.

If I had to maintain this in six months, I'd thank the author — the strict fixture validation means any future edit to the fixture that weakens the baseline will fail loudly, and the metric thresholds give a clear quality floor for the semantic recall work that follows.

中文说明

置信度:5/5 — 干净的纯测试 PR,完全按照 issue 要求实现,零生产风险。

这是一个直接、范围合理的评估基线。两个新文件,零生产代码变更,测试直接对 36 个合成多语言用例调用现有 selectRelevantChannelMemory 选择器。夹具契约足够严格以捕获漂移,指标是标准的(Recall@3、Top-1 准确率、无结果精确率),12 个关键用例的硬编码 requiredSelectedIds 提供了最强形式的回归保护。所有测试通过——6 个新评估测试、17 个现有 recall 测试和 496 个 ChannelBase 测试。Typecheck 和 build 均干净。

如果六个月后需要维护这个,我会感谢作者——严格的夹具校验意味着任何削弱基线的夹具修改都会大声报错,指标阈值为后续的语义召回工作提供了清晰的质量底线。

Qwen Code · qwen3.7-max

Reviewed at 0c19d1404f551537ba22bd65e87e41b0a8b6df47 · re-run with @qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review did not complete successfully. Qwen review aborted with an API error before posting comments. See workflow logs.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blockers. Suggestions are inline.

— qwen3.8-max-preview via Qwen Code /review

Comment on lines +440 to +446
for (const [caseId, expectedSelectedIds] of Object.entries(
requiredSelectedIds,
)) {
const testCase = cases.find((candidate) => candidate.id === caseId);
expect(testCase?.expectedSelectedIds).toEqual(expectedSelectedIds);
expect(selectedIdsByCase[caseId]).toEqual(expectedSelectedIds);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The requiredSelectedIds map and the set of fixture cases carrying expectedSelectedIds are only checked in one direction — this loop iterates the map, but nothing verifies that every fixture case with expectedSelectedIds also appears in it. The PR description promises a strict fixture contract where "exact ordering for critical cases" is "validated so future fixture drift cannot silently weaken the baseline," but a contributor who adds a new case with expectedSelectedIds (and updates requiredCaseIds so the load test passes) yet forgets the map entry would have that case's exact ordering silently unasserted — a later tie-breaking or budget-skip ordering regression on that case could ship undetected, since the statistical recall@3 / top-1 floors may not move on a single case. Concrete cost: the 12 current cases happen to match the 12 map keys, but that bijection is unenforced.

Suggested change
for (const [caseId, expectedSelectedIds] of Object.entries(
requiredSelectedIds,
)) {
const testCase = cases.find((candidate) => candidate.id === caseId);
expect(testCase?.expectedSelectedIds).toEqual(expectedSelectedIds);
expect(selectedIdsByCase[caseId]).toEqual(expectedSelectedIds);
}
for (const [caseId, expectedSelectedIds] of Object.entries(
requiredSelectedIds,
)) {
const testCase = cases.find((candidate) => candidate.id === caseId);
expect(testCase?.expectedSelectedIds).toEqual(expectedSelectedIds);
expect(selectedIdsByCase[caseId]).toEqual(expectedSelectedIds);
}
expect(
new Set(
cases
.filter((c) => c.expectedSelectedIds !== undefined)
.map((c) => c.id),
).size,
).toBe(Object.keys(requiredSelectedIds).length);

— qwen3.8-max-preview via Qwen Code /review

Comment on lines +181 to +185
if (relevantIds.length === 0) {
if (expectedTopId !== undefined) {
throw new Error(`${id}.expectedTopId is not allowed for no-result cases`);
}
} else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] parseCase rejects expectedTopId on no-result cases but applies no symmetric restriction to expectedSelectedIds, so a no-result case (relevantIds: []) carrying a non-empty expectedSelectedIds passes validation and becomes invisible to every metric: recallAt3 skips no-result cases, top1Accuracy skips cases without expectedTopId, and noResultPrecision only counts cases whose selection is empty (if a short entry qualifies for fallback selection, the selector returns non-empty and the case never enters the denominator). Concrete cost: the fixture could silently carry a semantically invalid no-result case that expects a selection, weakening the no-result coverage the baseline claims to freeze. The five current no-result cases are all pinned to [] via requiredSelectedIds, so this is a future-proofing gap rather than a present bug.

Suggested change
if (relevantIds.length === 0) {
if (expectedTopId !== undefined) {
throw new Error(`${id}.expectedTopId is not allowed for no-result cases`);
}
} else {
if (relevantIds.length === 0) {
if (expectedTopId !== undefined) {
throw new Error(`${id}.expectedTopId is not allowed for no-result cases`);
}
if (expectedSelectedIds && expectedSelectedIds.length > 0) {
throw new Error(
`${id}.expectedSelectedIds must be empty for no-result cases`,
);
}
} else {

— qwen3.8-max-preview via Qwen Code /review

@wenshao wenshao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Downgraded from Approve to Comment: CI failing: review-pr. Reviewed.

— qwen3.8-max-preview via Qwen Code /review

@wenshao

wenshao commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

✅ Local verification — real build & tests (maintainer review)

Built and ran this PR from a clean, isolated worktree with a full npm ci, then went beyond the pass/fail signal: I independently recomputed every metric against the actually-shipped compiled selector and injected regressions to prove the baseline is non-vacuous. Net result: approve to merge.

Environment

OS 🐧 Linux (x86-64)
Node / npm v22.22.2 / 10.9.7
Worktree head 222d9276a (merge of main into the PR branch)
Install npm ci (workspace, exit 0)
Scope test-only — exactly 2 files, +1074/-0, git diff confirms zero runtime change

1) Build + tests — 519 passed

New eval (6) + the existing recall unit suite (17) + the full ChannelBase suite (496), no regressions:

 RUN  v3.2.4 /root/git/pr7220/packages/channels/base
 ✓ src/channel-memory-recall.test.ts       (17 tests)  9ms
 ✓ src/channel-memory-recall-eval.test.ts  (6 tests)   18ms
 ✓ src/ChannelBase.test.ts                 (496 tests) 8237ms
 Test Files  3 passed (3)
      Tests  519 passed (519)

Static checks on the two PR files: tsc --noEmit ✅, eslint ✅, prettier --check ✅. GitHub CI is green.


2) Independent metric reproduction — against the shipped dist/ selector

I wrote a standalone harness that imports the compiled dist/channel-memory-recall.js (the code that actually ships) and recomputes each metric straight from the checked-in fixture — not trusting the test's own assertions:

============ PR #7220 metric reproduction (shipped dist/ selector) ============
total cases ................ 36    (assert 36)
positive (recall) cases .... 31    (assert 31)
top-1 cases ................ 31    (assert 31)
labeled no-result cases .... 5     (assert 5)
empty predictions .......... 5     (of which correct: 5)
------------------------------------------------------------------------------
Recall@3 ................... 1.0000  100.00%   (floor >= 0.90)  PASS
Top-1 accuracy ............. 1.0000  100.00%   (floor >= 0.85)  PASS
No-result precision ........ 1.0000  100.00%   (must == 1.00)   PASS
Max selected entries ....... 3       (budget <= 3)              PASS
Max selected code points ... 1200    (budget <= 1200)           PASS
------------------------------------------------------------------------------
Per-category recall@3 / top-1: english/chinese/japanese/korean/mixed/
fallback/ranking/budget all 100.00% ; no-result n/a (empty by design)
Ordered-selection / top-1 mismatches vs fixture expectations: 0

Every claimed number reproduces exactly, and the fixture's per-case expectedSelectedIds / expectedTopId all match the live selector output (0 mismatches). The budget-skip-middle case lands at exactly 1200/1200 code points — the byte-budget path is genuinely exercised, not just skirted.


3) Non-vacuity proof — inject regressions, confirm the eval turns RED

A baseline is only worth merging if it actually fails when the thing it guards breaks. I injected 5 regressions (3 into the selector, 2 into the fixture-contract) and restored to pristine after each (git diff clean):

# Injected regression Guard that fired Result
A reverse the stable tie-break (left.index-right.index → reversed) ordered-IDs: en-stable-tie flips 🔴 caught
B disable fallback (fallback = []) noResultPrecision 0.714≠1 + fallback-single 🔴 caught (2 tests)
C single CJK char matches (index+1<lenindex<len) none-single-cjk leaks ['single'] 🔴 caught
D tamper category count (english 6→5) fixture load: category-count contract 🔴 caught
E relevantIds → non-existent entry id fixture load: reference-integrity contract 🔴 caught

Note on C: the aggregate no-result precision stayed 1.0 (the leak makes a prediction non-empty rather than wrongly-empty), yet the explicit per-case expectedSelectedIds: [] assertion still caught it — the aggregate floor and the exact per-case assertions are genuinely complementary layers. The test file also unit-tests its own scoring harness (the hand-built noResultPrecision === 0.5 case), so the evaluator logic itself is covered.


4) One non-blocking observation (floor headroom)

Observed scores are a perfect 1.00 / 1.00, while the aggregate floors sit at 0.90 / 0.85, and exact per-case ordering is asserted for 7 of the 31 positive cases. So on the other 24 positive cases, the aggregate floors alone have slack: a single top-1 flip → 30/31 = 0.968 (≥ 0.85), and up to 3 full recall misses → 28/31 = 0.903 still clears 0.90 (a 4th, 27/31 = 0.871, is what finally trips it). That's expected for a "floor" and the PR explicitly frames it as a bounded synthetic baseline — just flagging that, if the goal is catching single-case recall/top-1 regressions on the non-exact cases, either nudging the floors closer to the observed 1.00 or asserting expectedTopId per-case for all positive cases would tighten the net. Not a merge blocker.


Verdict

LGTM — approve to merge. Test-only, zero runtime change, all suites + static checks + CI green, every advertised metric independently reproduced against the shipped selector, and the baseline is proven to fail on real regressions (selector and fixture-contract). It establishes a genuine, reproducible quality floor for channel memory recall.

Verification artifacts (harness + terminal screenshots) generated locally; happy to attach the PNGs inline if useful.

中文版(点击展开)

✅ 本地验证 —— 真实构建与测试(维护者审阅)

在干净、隔离的 worktree 中通过完整 npm ci 构建并运行了本 PR,并且没有停留在“测试通过”这一层:我独立地基于实际打包产物中的 selector 重新计算了每一项指标,并且注入回归以证明该基线不是空跑(non-vacuous)。结论:同意合并。

环境

操作系统 🐧 Linux (x86-64)
Node / npm v22.22.2 / 10.9.7
Worktree HEAD 222d9276amain 合并进 PR 分支)
安装 npm ci(workspace,退出码 0)
改动范围 仅测试 —— 恰好 2 个文件,+1074/-0git diff 确认运行时零改动

1)构建 + 测试 —— 519 通过

新增 eval(6)+ 既有 recall 单元测试(17)+ 完整 ChannelBase(496),无回归:

 ✓ src/channel-memory-recall.test.ts       (17 tests)
 ✓ src/channel-memory-recall-eval.test.ts  (6 tests)
 ✓ src/ChannelBase.test.ts                 (496 tests)
      Tests  519 passed (519)

两个 PR 文件的静态检查:tsc --noEmit ✅、eslint ✅、prettier --check ✅。GitHub CI 绿。

2)独立指标复算 —— 基于打包 dist/ selector

我写了一个独立 harness,导入编译后的 dist/channel-memory-recall.js(真正随包发布的代码),直接从提交的夹具重新计算每项指标,不依赖测试自身的断言:

total cases 36 / positive 31 / top-1 31 / no-result 5 / empty 5(correct 5)
Recall@3           1.0000  (floor >= 0.90)   PASS
Top-1 accuracy     1.0000  (floor >= 0.85)   PASS
No-result precision 1.0000 (must == 1.00)    PASS
Max entries 3 (<=3) PASS ; Max code points 1200 (<=1200) PASS
各类别 english/chinese/japanese/korean/mixed/fallback/ranking/budget 均 100%
与夹具期望的 有序选择 / top-1 不一致数:0

每个数字都被精确复现,夹具中每条用例的 expectedSelectedIds / expectedTopId 与真实 selector 输出完全一致(0 不一致)。budget-skip-middle 恰好落在 1200/1200 code point —— 预算裁剪路径被真正触发,而非绕过。

3)非空跑证明 —— 注入回归,确认 eval 变红

一个基线只有在“它所保护的行为被破坏时会失败”时才值得合并。我注入了 5 个回归(3 个到 selector,2 个到夹具契约),每次之后都恢复到干净状态(git diff clean):

# 注入的回归 触发的守卫 结果
A 反转稳定 tie-break 有序 ID:en-stable-tie 顺序翻转 🔴 捕获
B 禁用 fallback(fallback = [] noResultPrecision 0.714≠1 + fallback-single 🔴 捕获(2 项)
C 单个 CJK 字符可匹配 none-single-cjk 泄漏出 ['single'] 🔴 捕获
D 篡改类别数量(english 6→5) 夹具加载:类别数量契约 🔴 捕获
E relevantIds 指向不存在的条目 夹具加载:引用完整性契约 🔴 捕获

关于 C:聚合的无结果 precision 仍为 1.0(泄漏使预测变为“非空”而非“错误地为空”),但显式的逐用例 expectedSelectedIds: [] 断言仍然抓到了它 —— 聚合下限与逐用例精确断言是真正互补的两层。测试文件还对自身的评分 harness 做了单测(手写的 noResultPrecision === 0.5 用例),因此评估器逻辑本身也有覆盖。

4)一点非阻塞观察(下限余量)

实测得分为完美的 1.00 / 1.00,而聚合下限为 0.90 / 0.85,且逐用例精确顺序仅对 31 个正向用例中的 7 个 做了断言。因此在其余 24 个正向用例上,仅靠聚合下限存在余量:单个 top-1 翻转 → 30/31 = 0.968(≥ 0.85);最多 3 个完整 recall miss → 28/31 = 0.903 仍然通过 0.90(第 4 个 27/31 = 0.871 才会触发失败)。对“下限”而言这属预期,PR 也明确将其定位为有界的合成基线 —— 仅提示:若目标是抓住非精确用例上的单用例 recall/top-1 回归,可将下限上调至更接近实测 1.00,或对所有正向用例逐条断言 expectedTopId,以收紧这张网。非合并阻塞项。

结论

LGTM —— 同意合并。 仅测试、运行时零改动,所有测试套件 + 静态检查 + CI 全绿,每项宣称指标都基于打包 selector 独立复现,且基线已被证明会在真实回归(selector 与夹具契约两侧)下失败。它为 channel memory recall 建立了一个真实、可复现的质量下限。

@wenshao wenshao added the autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+) label Jul 19, 2026
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator Author

🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the autofix/takeover label (or comment @qwen-code /takeover stop) to release.

中文说明

🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。移除 autofix/takeover 标签(或评论 @qwen-code /takeover stop)即可释放。

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator Author

🤖 Could not address the latest feedback automatically (round 1/100). A human should take over this PR.

What I found before stopping:
Qwen failed during address-review: status 1.

See the Qwen Autofix agent step logs for model/tool output.

Run log: https://github.com/QwenLM/qwen-code/actions/runs/29718309973


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@wenshao
wenshao added this pull request to the merge queue Jul 20, 2026
Merged via the queue into QwenLM:main with commit e0e7ce7 Jul 20, 2026
80 checks passed
qqqys pushed a commit to qqqys/qwen-code that referenced this pull request Jul 21, 2026
…enLM#7247)

* fix(autofix): retry a model API error instead of stranding the PR

When the agent's qwen subprocess dies on a model-side [API Error]
(403 access denied, a 429 quota, a 5xx), run-agent.mjs wrote a
handoff/failure.md, so the handoff step treated it as an EVALUATED
handoff — it advanced the watermark and the next scan saw 'nothing
new', stranding the PR until a manual re-arm. But the agent never
actually evaluated the feedback; the model was unreachable.

QwenLM#7220 hit exactly this: fork-takeover engaged and ran the agent, the
model returned '[API Error: 403 Model access denied]' (the autofix key
lacks access to qwen3.8-max-preview), and the PR was left with an
advanced watermark that will not retry.

Fix, mirroring QwenLM#7229's no-output-crash handling:
- run-agent.mjs extracts a [API Error: 4xx/5xx] from the captured
  output tail, includes it in failure.md, and drops an
  marker file.
- The handoff step reads that marker and routes the failure to the
  sentinel-ts (retry) path — the watermark does NOT advance, so the
  next scan retries; the round still increments so a PERSISTENT model
  failure is bounded by MAX_ROUNDS. The headline names the model error
  and, on the final attempt, tells the maintainer to check the autofix
  model key/access and re-arm — instead of a generic crash message.

Tests: run-agent.mjs flags a model [API Error] (marker + failure.md)
and does NOT flag a generic failure; the handoff replay treats an
API-error handoff as sentinel|retry (not a watermark advance) with a
model-aware, cause-specific headline. 62/62 + 12/12.

* fix(autofix): scope + broaden the retryable model-API detection (review)

Addresses wenshao's review on QwenLM#7247:

- Behavioral (1): the agent-api-error marker was written on ANY non-zero
  exit whose output tail contained an API-error string — so a loop
  guard, a timeout, or an agent-written failure.md (a real verdict)
  would wrongly retry and, worst case, silently discard a verdict. The
  write is now scoped to the bare-failure branch and guarded by
  !timedOut, so only an un-evaluated model failure retries.
- Coverage (2): the old regex only matched a LEADING status digit, so
  it missed the canonical rate-limit render, the (Status: …) form, the
  bad-key 401, the Chinese quota text, and the unwrapped Qwen OAuth
  quota — i.e. most real errors this targets. Detection is now a
  whitelist of RECOVERABLE errors (401/402/403/429/5xx + rate-limit /
  quota / api-key / RESOURCE_EXHAUSTED / overloaded phrasings, plus the
  standalone OAuth-quota form); a 400/404 stays terminal.
- Test gap (3): a writer↔reader contract test now runs the REAL
  run-agent.mjs to write the marker, then the extracted workflow reader
  block against that same workdir — a rename on either side (proven
  with the YAML-only mutation) now fails the suite.
- Smaller: API_ERROR_DETAIL is comment-escaped (sed) and capped
  (cut -c1-200) since it derives from agent stdout; the marker match is
  single-line ([^]\n]) so a multi-line render can't smuggle a newline;
  agent-api-error is added to the run-artifacts list.

Non-recoverable 4xx (400/404) deliberately stay terminal; the live
401/403 config cases retry and self-heal once the key/access is fixed.
79/79 across both suites.

* test(autofix): cover the timeout guard and the OAuth-quota fallback (review)

Two coverage gaps from the ci-bot review on QwenLM#7247:
- The !result.timedOut guard was only asserted indirectly — no test
  emitted an [API Error] AND timed out. Added a case (spawnSync +
  QWEN_TIMEOUT_MS=100): qwen streams [API Error: 503] then hangs past
  the budget → killed → no marker. A refactor to !loopDetected now
  fails here.
- The standalone Qwen-OAuth-quota fallback (unwrapped, no [API Error:])
  had no test. Added a case emitting bare 'Qwen OAuth quota exceeded
  (limit: 100/min)' → marker written, wrapped as
  '[API Error: Qwen OAuth quota exceeded …]'.

* fix(autofix): anchor the API-error code, split retry budget by cause, keep the headline UTF-8

Addresses the review on QwenLM#7247.

Classifier (points 2 and 4): the status code is now read from its POSITION in
the render (`[API Error: <code>`) instead of matched anywhere in the message.
Matching anywhere retried permanent failures forever — `400 Invalid value for
max_tokens: must be <= 512` matched a bare \b5\d\d\b and `400 context length
exceeded` matched a bare `exceeded`. `exceeded` now only counts as part of
`quota`. A 404 whose message says the model "does not exist or you do not have
access to it" — the OpenAI-compatible render of what a 403 reports — is no
longer terminal.

Retry budget (point 3): the marker now carries the cause class. A transient
429/5xx self-heals and keeps the full round budget; an auth/access error that
only a maintainer can fix is capped at API_AUTH_MAX_ROUNDS (3) and then goes
terminal with the "check the autofix model key/access, then re-arm" headline —
instead of ~100 agent runs and ~100 PR comments over ~17h on a takeover PR.
The terminal round is stamped so the scan's round gate skips the PR while the
sentinel ts keeps the feedback live for a re-arm.

Headline (point 1): `cut -c` counts bytes under GNU coreutils and the
classifier deliberately matches CJK renders, so the 200-byte cap could split a
multi-byte character and emit invalid UTF-8. Guarded with
`iconv -f utf-8 -t utf-8 -c || true`, matching the sibling publish site (the
`|| true` is required — iconv -c exits 1 when it discards).

Minor (point 5): documented that detection is best-effort because apiError is
derived from the last 20 KB of output; `head -1` -> `head -n 1`; tests added
for a permanent 400 carrying a 3-digit number >= 500 and for a >200-byte CJK
render staying valid UTF-8.

* test(autofix): cover the auth-capped retry budget and Chinese API-error patterns (QwenLM#7247)

* fix(autofix): short-circuit 400 as terminal and classify only the last API error (QwenLM#7247)

* fix(autofix): treat transport-level API failures as retryable

QwenLM#7365 stranded at round 2/100 on this render:

    [API Error: terminated (cause: read ECONNRESET)]

The connection to the model dropped mid-run. That is as transient as a 429, but
the classifier never saw it that way: a transport failure never got far enough
to have an HTTP status, so it fell through to the keyword arm, and the keyword
arm only knew about rate limits and quotas. It was classified terminal, the
watermark advanced, and a PR that needed nothing but a re-run was handed to a
human.

Verified against the shipped classifier before the fix — every transport render
came back terminal:

    terminated (cause: read ECONNRESET)   -> terminal
    fetch failed                          -> terminal
    socket hang up                        -> terminal
    connect ETIMEDOUT                     -> terminal

Adds a transport arm to the code-less branch: ECONNRESET, ECONNREFUSED,
ETIMEDOUT, EPIPE, EAI_AGAIN, socket hang up, fetch failed, terminated.

ENOTFOUND is deliberately excluded. A hostname that does not resolve is a
misconfigured endpoint, which repeats forever — the same reasoning that keeps a
bad model name terminal.

Coded errors are unaffected: the arm sits after the status-code branch, so the
400 short-circuit added in 719991a still runs first.

* fix(autofix): address review — OAuth fallback override, comment accuracy, display clamp (QwenLM#7247)

---------

Co-authored-by: wenshao <wenshao@example.com>
Co-authored-by: 易良 <1204183885@qq.com>
Co-authored-by: qwen-code-ci-bot <qwen-code-ci-bot@users.noreply.github.com>
@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Released in v0.20.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a multilingual evaluation baseline for channel memory recall

4 participants