feat(core): add memory recall delivery telemetry - #7393
Conversation
E2E Test ReportRan tmux-based CLI E2E against the rebased branch in isolated project Overall result: PASS, with 3/5 scenarios fully verified and 2 scenarios timing-limited in real TUI. Verified scenarios:
Timing-limited scenarios:
Telemetry checks:
Full local report: |
|
Thanks for the PR! Template looks good ✓ Problem: This is the first of three PRs from RFC #7040 (narrowed by the Core memory maintainer). The gap is real and observed: recall selection telemetry exists, but there's no signal for whether selected memories actually reached the model prompt, arrived late at a ToolResult continuation, or were silently dropped. That makes it impossible to evaluate delivery behavior before changing the recall pipeline. Direction: Aligned — issue #7040 is on the Size: 256 production lines (client.ts 126, loggers.ts 36, metrics.ts 51, types.ts 39, constants.ts 1, index.ts 3) vs. 837 test lines (client.test.ts 733, loggers.test.ts 78, metrics.test.ts 26). Well under the 500-line threshold — no escalation needed. Approach: Scope feels right. Every edit serves the stated goal — new event type, logger, metrics, and wiring terminal outcomes into the existing prefetch lifecycle. The Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题： 这是 RFC #7040（由 Core memory 维护者收窄后）的三个 PR 中的第一个。差距是真实且已观察到的：recall selection telemetry 已存在，但没有信号表明选中的 memory 是否真正进入了模型 prompt、是否在 ToolResult continuation 点晚到投递、还是被静默丢弃。缺少这个信号就无法在修改 recall 管道之前评估投递行为。 方向： 对齐——issue #7040 在 规模： 256 行生产代码 vs. 837 行测试代码。远低于 500 行阈值，无需升级。 方案： 范围合理。每个编辑都服务于既定目标。 进入代码审查 🔍 — Qwen Code · qwen3.7-max Reviewed at |
Code ReviewIndependent proposal: Given the goal (add delivery/discard telemetry for auto-memory recall), I would: add a Comparison with the diff: The PR matches this proposal almost exactly. The implementation follows existing telemetry patterns ( Key observations:
No blockers found. E2E Testingtmux is not available on this CI runner, so I ran a direct headless CLI test with telemetry outfile export. Setup: seeded CLI output (PR code): Telemetry outfile — delivery event: {
"event.name": "qwen-code.memory.recall.delivery",
"event.timestamp": "2026-07-22T01:27:58.038Z",
"phase": "refined",
"delivery_point": "tool_result",
"strategy": "heuristic",
"docs_selected": 1,
"latency_ms": 5323
}Telemetry outfile — delivery metrics: ✅ Memory recalled and delivered correctly. Delivery event emitted with one terminal outcome ( Unit tests: Build + typecheck: both pass. 中文说明代码审查独立方案： 给定目标（为 auto-memory recall 添加投递/丢弃遥测），我会：在 与 diff 对比： PR 几乎完全匹配此方案。实现遵循现有遥测模式。未发现正确性 bug、安全漏洞或回归。 关键观察：
无阻塞问题。 E2E 测试此 CI 运行器无 tmux，改用直接无头 CLI 测试。CLI 正常响应，delivery 事件和指标正确发出，属性仅含低基数字段，无 PII 泄漏。 单元测试：3 个文件 390 个测试全部通过。构建和类型检查均通过。 — Qwen Code · qwen3.7-max Reviewed at |
|
Confidence: 4/5 — clean implementation that mirrors existing telemetry patterns exactly, thorough test coverage, verified E2E; the only non-blocking nit is that Honest take: this is a well-scoped, well-tested telemetry addition. The implementation does exactly what my independent proposal would have done — same event type, same logger pattern, same metrics structure, same terminal-event guard. The E2E confirms the event fires with the right attributes and no PII leakage. 390 unit tests pass, build and typecheck are green. Every change in the diff serves the stated goal — no scope creep, no drive-by refactors. If I had to maintain this in six months, I'd thank the author: the terminal-event accounting is easy to follow, the typed discard reasons make debugging straightforward, and the test coverage means I can refactor the prefetch lifecycle without fear. 中文说明置信度：4/5 —— 实现干净，完全镜像现有遥测模式，测试覆盖全面，E2E 已验证；唯一的非阻塞小问题是 诚实评价：这是一个范围合理、测试充分的遥测补充。实现与我的独立方案完全一致。 E2E 确认事件以正确属性触发，无 PII 泄漏。390 个单元测试通过，构建和类型检查绿色。diff 中每个变更都服务于既定目标——无范围蔓延，无顺手重构。 — Qwen Code · qwen3.7-max Reviewed at |
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. 1 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here.
— qwen3.7-max via Qwen Code /review
| this.cancelPendingMemoryPrefetch('shutdown'); | ||
| } |
There was a problem hiding this comment.
[Suggestion] requestShutdown() calls cancelPendingMemoryPrefetch('shutdown') — a new discard reason — but no test covers this path. The existing requestShutdown tests verify background memory task gating but none set up a pending prefetch handle, so the cancel path is never exercised in tests.
Failure scenario: A future refactor of requestShutdown() could remove or misplace the cancelPendingMemoryPrefetch('shutdown') call without any test detecting the regression. Operators would lose telemetry for this terminal outcome.
Suggested fix: Add a test that sets up a pending prefetch, calls requestShutdown(), and asserts logMemoryRecallDelivery was called with discard_reason: 'shutdown'. Pattern after the existing reset discard test.
— qwen3.7-max via Qwen Code /review
| if (signal.aborted) { | ||
| controller.abort(); | ||
| } else { |
There was a problem hiding this comment.
[Suggestion] When the parent signal is already aborted at handle creation, only controller.abort() fires — cancelPendingMemoryPrefetch('abort') is not called. The onParentAbort handler in the else branch includes both calls, but the pre-aborted path only mirrors the first.
Failure scenario: A caller enters with an already-aborted AbortSignal. The recall controller is aborted and the catch handler returns EMPTY_RELEVANT_AUTO_MEMORY_RESULT, but the delivery event won't carry discard_reason: 'abort' — operators cannot distinguish pre-abort discard from other empty-result discards.
Suggested fix: Mirror the onParentAbort handler's this.cancelPendingMemoryPrefetch('abort') call in the pre-aborted branch. Note the handle may not yet be installed on this.pendingMemoryPrefetch at this point, so this may require restructuring.
— qwen3.7-max via Qwen Code /review
| export type MemoryRecallDeliveryPhase = 'fast' | 'refined'; | ||
| export type MemoryRecallDeliveryPoint = 'initial' | 'tool_result' | 'discarded'; | ||
| export type MemoryRecallDiscardReason = | ||
| | 'not_ready' |
There was a problem hiding this comment.
[Suggestion] 'not_ready' is declared as a MemoryRecallDiscardReason variant and mirrored in metrics.ts, but no caller in client.ts ever passes this value — all cancel sites use 'shutdown', 'reset', 'new_query', 'abort', 'no_safe_delivery_point', or 'superseded'.
Failure scenario: The dead variant inflates the type union without exercising any code path. A future reader may think there's a code path that produces 'not_ready', or a linter exemption may accumulate around it.
Suggested fix: Remove 'not_ready' from MemoryRecallDiscardReason in both types.ts and metrics.ts, or add it when a caller needs it.
— qwen3.7-max via Qwen Code /review
| if (handle.terminalLogged) return; | ||
| handle.terminalLogged = true; |
There was a problem hiding this comment.
[Suggestion] The terminalLogged idempotency guard is not directly tested. No test constructs a scenario where a handle's terminalLogged is set true before a second call to logMemoryPrefetchDelivery.
Failure scenario: A regression that removes the guard would go undetected by the test suite until a rare timing-dependent double-log surfaces in production (e.g., a race between cancelPendingMemoryPrefetch and tryConsumeMemoryPrefetch).
Suggested fix: Add a test that calls logMemoryPrefetchDelivery (or triggers both delivery and discard) twice on the same handle and asserts logMemoryRecallDelivery is called exactly once.
— qwen3.7-max via Qwen Code /review
| recordMemoryRecallDeliveryMetrics(config, event.latency_ms, { | ||
| phase: event.phase, | ||
| delivery_point: event.delivery_point, | ||
| discard_reason: event.discard_reason, | ||
| strategy: event.strategy, | ||
| docs_selected: event.docs_selected, | ||
| }); |
There was a problem hiding this comment.
[Suggestion] logMemoryRecallDelivery conditionally includes discard_reason in log attributes (if (event.discard_reason)) but unconditionally passes event.discard_reason (which may be undefined) to recordMemoryRecallDeliveryMetrics. The metrics function compensates internally with its own conditional spread, but the caller→callee contract is inconsistent.
Failure scenario: On every non-discard delivery (delivery_point: 'initial' or 'tool_result'), the metrics function receives { discard_reason: undefined }. If a future refactor removes the internal guard (relying on callers to omit the key), metric attributes would silently gain discard_reason: undefined as a tag value, polluting dashboard grouping.
| recordMemoryRecallDeliveryMetrics(config, event.latency_ms, { | |
| phase: event.phase, | |
| delivery_point: event.delivery_point, | |
| discard_reason: event.discard_reason, | |
| strategy: event.strategy, | |
| docs_selected: event.docs_selected, | |
| }); | |
| recordMemoryRecallDeliveryMetrics(config, event.latency_ms, { | |
| phase: event.phase, | |
| delivery_point: event.delivery_point, | |
| ...(event.discard_reason ? { discard_reason: event.discard_reason } : {}), | |
| strategy: event.strategy, | |
| docs_selected: event.docs_selected, | |
| }); |
— qwen3.7-max via Qwen Code /review
yiliang114
left a comment
yiliang114
left a comment
There was a problem hiding this comment.
Reviewed the delivery/discard lifecycle independently — the terminal-event accounting is sound.
- The
terminalLoggedguard plus the synchronousconsumed = true/pendingMemoryPrefetch = undefinedmarking at the top oftryConsumeMemoryPrefetch(before theawait) closes the consume↔cancel race. Traced consume→cancel, parent-abort→cancel, and the pre-aborted-signal paths; each lands exactly one terminal event. Thefinallybelt-and-suspenders at the bottom ofsendMessageStreamcovers the early-return sites. - No terminal event is lost on the normal path: a settled-but-unconsumed prefetch is deliberately preserved for the next ToolResult consume and is always closed out by the next query / reset / shutdown / abort.
- Cardinality is right —
docs_selected(unbounded) stays on the log event but is excluded from the metric attributes, andsession.idstays opt-in.
The five suggestions already on the PR are valid non-blocking; nothing to add on top. The pre-aborted-signal one is real but narrow: the finally cleanup applies discard_reason: 'abort' unless an await between prefetch setup and the consume point lets the aborted recall settle and get consumed first.
|
Independently traced the delivery/discard lifecycle end-to-end — no new blockers, the terminal-event accounting holds up. One non-blocking follow-up worth tightening: Not blocking — fine to land as-is. |
8236edd to
69c925e
Compare
|
Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. 中文请勿对活跃的 PR 执行 rebase 或 force-push，因为这会使已有的评审评论失效。另外，供日后参考：作为集成流程的一部分，机器人始终会自动将所有改动压缩（squash）为单个提交。 |
yiliang114
left a comment
yiliang114
left a comment
There was a problem hiding this comment.
Found one terminal-accounting issue that looks worth fixing before merge.
| // future early-return sites that forget to call cancel. | ||
| if (!normalCompletion) { | ||
| this.cancelPendingMemoryPrefetch(); | ||
| this.cancelPendingMemoryPrefetch( |
There was a problem hiding this comment.
[P1] A slow recall can still miss terminal delivery telemetry on a normal no-tool turn. This cleanup only runs when normalCompletion is false; when recall is still pending at the initial consume point and the model finishes without tool calls or another continuation, the bottom-of-try path sets normalCompletion = true and preserves the handle for a future ToolResult. But no ToolResult will be scheduled for that no-tool turn, so the prefetch has no terminal delivery/discard event until a later unrelated new query/reset logs a misleading reason, or never if the session ends. That violates the PR's one-terminal-outcome goal; the no-future-delivery path should close the pending prefetch as no_safe_delivery_point (or equivalent) before returning.
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
[Critical] client.ts:3074 — Confirmed (verified): on a normal no-tool turn, normalCompletion = true preserves a still-pending memory prefetch for a future ToolResult that will never arrive. The finally cleanup (line 3089) only fires when !normalCompletion, so no terminal delivery/discard event is emitted. The handle is orphaned until the next unrelated cancel (new_query, reset) logs a misleading discard_reason with inflated latency, or never closed if the session ends. Violates the PR's one-terminal-outcome-per-prefetch goal. (Existing P1 from @yiliang114 at client.ts:3071, confirmed still-standing.)
— qwen3.7-max via Qwen Code /review
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
On a normal turn where the model responds without tool calls, the pending memory prefetch was preserved for a future ToolResult that would never arrive. This orphaned the handle until an unrelated cancel logged a misleading discard_reason with inflated latency, or never closed it if the session ended. Track whether the streaming loop saw any ToolCallRequest events and cancel the prefetch with no_safe_delivery_point at normalCompletion when none were requested. Also reuse the shared delivery/discard type aliases in metrics.ts instead of re-declaring them inline.
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: Review Feedback SummaryRequired fixes (Critical / P1)Orphaned prefetch on no-tool turns ([Critical] @qwen-code-ci-bot at client.ts:3074, [P1] @yiliang114 at client.ts:3071)
Suggestions implementedReuse shared type aliases in metrics.ts (@yiliang114 issue-level comment)
Suggestions declined (with reasons)[rc:3620184792] Add test for
[rc:3620184796] Pre-aborted signal path missing
[rc:3620184804] Remove dead
[rc:3620184810] Test the
[rc:3620184815] Inconsistent
Verification
Conflict notesNo conflicts ( 中文说明审查反馈总结必须修复（Critical / P1）无工具调用回合中孤立的预取句柄（[Critical] @qwen-code-ci-bot 在 client.ts:3074，[P1] @yiliang114 在 client.ts:3071）
已实施的建议在 metrics.ts 中复用共享类型别名（@yiliang114 issue 级评论）
已拒绝的建议（附原因）[rc:3620184792] 为
[rc:3620184796] 预中止信号路径缺少
[rc:3620184804] 移除死代码
[rc:3620184810] 测试
[rc:3620184815]
验证
冲突说明无冲突（ Base-conflict check: no conflict with main. Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
— qwen3.7-max via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 2/100). What changed, and what I pushed back on: Review feedback addressedAll four suggestions from the automated reviewer have been implemented. 1. Add
|
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
Reviewed. Not reviewed: You are review agent verify — Verification agent (round... — pointed at diff lines it never opened: it made tool calls, but none of them read the diff.
— qwen3.7-max via Qwen Code /review
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
…nd arena cancel telemetry (QwenLM#7393)
|
🤖 Addressed the latest review feedback (round 3/100). What changed, and what I pushed back on: Review feedback addressedrc:3625266545 —
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. 2 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here. Not reviewed: issue-fidelity — Agent 0 ran and returned a substantive fidelity assessment but the harness did not match its launch prompt to the CLI-built brief. Not reviewed: build-and-test — Agent 7 ran build (16 workspaces passed) and tests (388 PR-changed tests passed) but the harness did not match its launch prompt to the CLI-built brief. Not reviewed: Agent 0: Issue fidelity & root-cause ownership, Agent 7: Build & test verification — its prompt was built, but no agent on record was launched with it.
— qwen3.7-max via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 4/100). What changed, and what I pushed back on: Review feedback addressed[rc:3625927519] Add
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 5/100). What changed, and what I pushed back on: Review feedback addressed[rc:3626340356]
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
— qwen3.7-max via Qwen Code /review
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: Autofix review triage — PR #7393No code changes were made in this round. Feedback reviewed
Triage result
Since there is nothing actionable, the PR head is left unchanged. No commit was 中文说明Autofix 审查分诊 — PR #7393本轮未做任何代码改动。 已审阅的反馈
分诊结果
由于没有可处理的内容，PR 的 head 分支保持不变。未创建任何提交，也没有需要 Base-conflict check: no conflict with main. 🧠 Handled by Qwen Code · model/模型 |
|
@qwen-code /triage |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
✅ Local real-build verification (Linux) — merge referenceVerified at HEAD 1. Reviewer test plan — unit tests ✅The 15 new delivery-telemetry cases (12 in 2. Build + typecheck ✅
3. Negative control (RED/GREEN) ✅Reverting only 4. Real end-to-end on Linux ✅ (the part the PR couldn't run)Drove the real bundled CLI (
Not reproduced in a live TUI (timing, exactly as you noted): the Screenshots Real TUI run —
Delivery records + metrics from the real telemetry outfile (all three outcomes):
Verdict: LGTM from a verification standpoint — behavior matches the description, the delivery/discard signal is low-cardinality and PII-free, and the terminal-event guard is solid on Linux end-to-end. 中文说明✅ 本地真实构建验证（Linux）— 合并参考在 HEAD 1. Reviewer test plan — 单元测试 ✅本 PR 新增的 15 个投递遥测用例（ 2. 构建 + 类型检查 ✅
3. 负对照（RED/GREEN）✅只把 4. Linux 上的真实端到端 ✅（PR 未能覆盖的部分）在伪终端下驱动真实打包后的 CLI（
真实 TUI 中未复现（时序原因，正如你所说）： 截图 真实 TUI 运行——
真实遥测 outfile 中的投递记录 + 指标（三种结局）：
结论： 从验证角度 LGTM——行为与描述一致，投递/丢弃信号低基数且无 PII，终态事件守卫在 Linux 端到端下稳固。 Verified locally with Claude Code (Opus 4.8, 1M context) — real bundled-CLI E2E + focused unit tests on Linux. |
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: No action taken on PR #7393The only new feedback since the last evaluation is an issue-level comment from Triage
There are no reviews (no 中文说明PR #7393 未采取任何操作自上次评估以来唯一的新反馈是 @wenshao 的一条 issue 级评论：一份本地真实构建验证报告（Linux），并以明确的 LGTM 结尾。该评论不包含任何改动请求、未指名任何缺陷，也没有任何建议。 分类
没有 review（无 Base-conflict check: no conflict with main. 🧠 Handled by Qwen Code · model/模型 |
memory-system.md documented recall selection but never documented delivery, so the delivery telemetry from #7393 was undocumented and the fast path had no home in the canonical reference. Add a delivery section and the delivery event table, and correct two docs that still described the single-path behaviour.
…M#8716) * fix(memory): improve recall delivery and multilingual fallback * fix(memory): bound heuristic recall scoring * test(memory): pin initial recall budget with fake timers Rewrite the slow-recall test to assert with fake timers that the main request is still held 1 ms inside the 100 ms initial budget and proceeds without memory at expiry, so budget changes can no longer pass unnoticed. * test(memory): pin recall budget and scoring contracts Address review findings with mutation-verified pins: - settle-early: bounded wait ends when recall settles, not at full budget - Cron and ToolResult consume points stay zero-wait - post-wait replacement guard refuses stale handles - type boost flips the winner (tie-break no longer masks its removal) - hiragana-only coverage for the CJK tokenizer - design doc: RFC QwenLM#7040 sets no numeric overhead target; fix attribution * fix(memory): preserve recall field weighting * fix(memory): recall relevant topics beyond scan cap (QwenLM#8803) * fix(memory): bound recall candidates after full scan * test(memory): pin bounded selector inputs * fix(memory): preserve bounded recall candidates * fix(memory): preserve lexical recall candidates * fix(memory): prioritize lexical model candidates * fix(core): preserve UTF-16 manifest boundaries * fix(memory): address recall review feedback * test(memory): measure recall rollout gate against the pre-change scorer RFC QwenLM#7040 gates the multilingual precision change on evidence that English Recall@5 and no-result precision do not regress. Add a labeled 45-case corpus and an evaluation harness that scores both the shipped deterministic selector and a frozen copy of the pre-change scorer over it, so the gate is reproducible rather than asserted. * fix(memory): deliver a deterministic fast recall result on the initial turn The initial-turn budget is 100 ms, but recall awaits the model selector, which is a network side query with a 30 s ceiling. The budget therefore expires on the common path and delivery falls through to the ToolResult point — which a tool-free turn never reaches, so the result is discarded as no_safe_delivery_point. That is the case memory matters most for. Publish the deterministic candidates that selectModelCandidateDocuments already computes, before blocking on the selector, and inject them when the budget expires. The refined result still lands at ToolResult, with documents the fast phase already delivered filtered out. phase telemetry now carries both stages: phase is the delivery stage, strategy is the selection method, and they are orthogonal. * docs(memory): record the fast-path decision and phase/strategy split * test(memory): report the mixed-language slice in the rollout gate * docs(memory): align recall docs on the deterministic fast path memory-system.md documented recall selection but never documented delivery, so the delivery telemetry from QwenLM#7393 was undocumented and the fast path had no home in the canonical reference. Add a delivery section and the delivery event table, and correct two docs that still described the single-path behaviour. * fix(memory): use Array<T> for the fast-path test doc lists @typescript-eslint/array-type forbids T[] for non-simple types. * docs(memory): clarify recall delivery telemetry * fix(memory): report already-delivered recall count * docs(memory): align recall delivery claims * fix(memory): rank ties by recency and record fast-delivered discards Three review follow-ups on the recall reliability change. Tie-break: `selectRelevantAutoMemoryDocuments` broke score ties with `type.localeCompare`, which orders feedback < project < reference < user. That was tolerable while the result was five documents wide; the fast path takes only MAX_FAST_RECALL_DOCS = 2, so a tied user-typed document was dropped every time — the exact memory a tool-free turn exists to surface. Ties now fall to recency, then to input order, which keeps the project-before-user precedence the concatenation already establishes. Corpus: the case labeled `semantic-no-lexical` had no relevant documents, so it was a no-result case wearing the wrong label and nothing measured the cost of "no lexical match, no score". Relabel it and add three genuine answerable-but-lexically-disjoint cases. Both scorers return nothing for them, so the slice sits outside the quality floor and is asserted separately: the fast path closes the timing gap, not the matching gap. Tool-free delivery is 92.3%, not 100%, and the residual is that slice. Telemetry: a tool-free turn logs its terminal event from the discard path, which did not apply the fast-phase exclusion. A turn whose every selected document had already been fast-delivered was recorded as `no_safe_delivery_point`, inflating the "memory never reached the model" bucket with turns that got it. Apply the same rule the ToolResult consume point uses; a partial overlap still reports the cancellation reason. * docs(memory): state the candidate-cap trade and the per-turn document count Two review follow-ups, documentation only. No behaviour change. "Removes the 200-document cap" oversold the candidate change. What it does is swap a per-scope, query-blind recency truncation for a global, query-aware one, and the effect is not a uniform widening: at or under 200 documents nothing was excluded by count under either design, but the new 25,000-byte manifest budget is a ceiling the old path lacked; between 200 and 400 with neither scope over 200 the old path sent every document and the new one sends at most 200, so fewer reach the model; only a scope over 200 is the case the change is actually for. Record all three, plus the fact that the manifest budget packs rather than prefixes. MAX_RELEVANT_DOCS = 5 bounds one prompt, not one turn. A fast delivery of two plus a refined delivery of five disjoint documents puts seven in front of the model; dedupe removes repeats, not the sum. This follows from dropping combined fast/refined budget accounting, which was a deliberate choice, but the number was never written down next to the constant that reads like a hard cap. * fix(memory): end the initial recall wait on the fast result, widen tokenization The 100 ms initial budget was a fixed cost, and the evidence for it measured the wrong thing. Deterministic *scoring* is microseconds, but the fast result is only published once recall has enumerated, read, and parsed the memory tree — and this branch removed the 200-document cap for recall, so that scan grows with the tree. recall-scan-latency.test.ts adds that measurement against a real temporary tree: ~29 ms at 200 topics, ~70 ms at 500, ~130 ms at 1000. So for any tree small enough to scan in time — the ordinary case — the fast result was in hand tens of milliseconds before the budget expired, and the rest of the budget was spent waiting on a model selector this design already assumes will miss it. The wait now ends on whichever comes first: recall settling, the fast result being published, cancellation, or the ceiling. The preference order is unchanged, because the code after the wait still prefers a settled recall. Past roughly a thousand topics the scan alone exceeds the ceiling and the turn pays the full budget for nothing; that is recorded as a known limitation rather than fixed, since the fix is a persistent catalog. Tokenization kept only [a-z0-9]{3,} runs, so Cyrillic, Greek, Arabic, and accented Latin produced no tokens at all and the deterministic path was unconditionally silent for them. Keep whole runs of non-CJK letters, marks, and digits instead. CJK is excluded per character rather than by alternation order: \p{L} also matches Han, so a Latin-initial run would otherwise swallow the CJK after it and turn abc漢字 into one token. Scripts without word separators outside the CJK set still collapse to one run, which is recorded rather than claimed as segmentation. Two smaller follow-ups. The active-tool alias set is now derived once per recall instead of once per scanned document, which mattered little under the old 200-document cap and more without it. And the eval prints the Recall@5 a query-blind random scorer would score on this corpus (20%), with a test holding that floor at or below 25%, because a small corpus flatters every design and the headline was unreadable without it. * docs(memory): correct the initial-turn preference claim, pin it with a test Local end-to-end verification on QwenLM#8716 found the claim added in 01ef7d7 — "the preference order is unchanged: whatever ends the wait, a settled recall is still delivered in preference to the fast result" — to be false in the case that matters. `onFastResult` is published before recall issues the selector request at all, so the recall promise cannot be settled when the wait ends on the fast result. Measured against a selector settling in 15 ms, comfortably inside the ceiling, the initial turn still delivers the deterministic pair and discards the model's picks. The behaviour is right and stays: a model side query does not return inside a 100 ms ceiling in production, so arbitrating would spend the rest of the budget on every turn to win a race that does not happen, and the selector's judgement still lands at ToolResult with the fast documents excluded. What was wrong was the description. State it directly instead — on the initial turn, once the deterministic scorer matches, the fast result wins regardless of selector latency — and pin it with a test that fails when the early exit is removed, so it reads as a decision rather than an accident. Two measurements corrected while here. The scan crossover is machine- dependent, not a fixed topic count: the same three sizes measure 9/21/46 ms on faster hardware against 29/70/130 ms on the machine the tables were written from, so the ceiling is not reached there at all. And MAX_MODEL_CANDIDATE_DOCS = 200 is rarely the binding constraint — MAX_MODEL_MANIFEST_BYTES is, at roughly 90-150 documents once absolute paths and timestamps are counted. Measured runs sent 94 and 96 manifest lines where the document cap would have allowed 200, which also explains why the recency reserve has to be interleaved rather than appended. --------- Co-authored-by: yiliang114 <jinjing.zzj@gmail.com> Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
6 participants
What this PR does
Adds terminal delivery telemetry for managed auto-memory recall. The existing recall selection telemetry tells us which memories were selected, but not whether those selected memories were actually delivered to the main model. This PR adds a new
qwen-code.memory.recall.deliveryevent plus count and latency metrics, and records terminal outcomes when recall is injected into the initial prompt, injected at a ToolResult continuation point, discarded because no memory was selected, or cancelled by reset, shutdown, abort, new query, or another no-safe-delivery-point exit.The current single auto-memory prefetch path is reported as
phase: "refined"to preserve the current behavior and leave room for the planned Fast/Refined split in follow-up PRs. This PR does not change recall selection behavior or introduce Fast/Refined delivery logic.Why it's needed
Issue #7040 needs reliable observability for auto-memory recall delivery. Today we can see that recall selected memories, but we cannot tell whether those memories reached the model prompt, arrived later at a safe continuation point, or were dropped because the turn ended first. That makes it hard to evaluate first-turn delivery, late delivery, and cancellation behavior before changing the recall pipeline.
This PR adds the missing delivery/discard signal with low-cardinality fields only. The event intentionally does not include query text, query hashes, memory content, file paths, project paths, session/message ids, raw errors, or secrets.
Reviewer Test Plan
How to verify
Run the focused telemetry and client tests. Confirm that delivery telemetry is emitted for initial prompt injection, ToolResult injection, empty recall results, reset cancellation, and new-query supersession, and that metrics only carry low-cardinality attributes.
For E2E, run the CLI in a tmux session with
QWEN_CODE_MEMORY_LOCAL=1and telemetry file export enabled. Seed a local.qwen/memory/file with a unique keyword such asultraviolet parsnip, ask a prompt that recalls that memory, and inspect the telemetry outfile forqwen-code.memory.recall.delivery,qwen-code.memory.recall.delivery.count, andqwen-code.memory.recall.delivery.latency. Confirm each delivery event has one terminal outcome and does not include query, memory content, paths, session/message ids, raw errors, or secrets.Evidence (Before & After)
Before: auto-memory recall telemetry reported selection outcome only, so a selected memory could not be distinguished from a delivered, late-delivered, or discarded memory.
After: tmux E2E on
/tmp/pr1-memory-testproducedqwen-code.memory.recall.deliveryevents and delivery metrics. Verified normal recall, ToolResult delivery, and abort discard.new_queryandresetwere timing-limited in real TUI because recall completed before cancellation, but those cancellation paths are covered by unit tests.Tested on
Environment (optional)
Local worktree on macOS, built bundle and ran focused Vitest tests plus tmux-based CLI E2E with telemetry outfile export.
Risk & Scope
new_queryandresetdiscard reasons due to timing, but unit tests cover them.Linked Issues
Refs #7040
中文说明
这个 PR 做了什么
这个 PR 为 managed auto-memory recall 增加终态投递遥测。现有 recall selection telemetry 能说明选中了哪些 memory，但不能说明这些 memory 是否真的送进了主模型。这个 PR 新增
qwen-code.memory.recall.delivery事件，以及 count 和 latency metrics，并在 recall 注入首轮 prompt、注入 ToolResult continuation、安全点缺失导致丢弃、reset、shutdown、abort、新 query 顶替等路径记录终态。当前代码只有单一 auto-memory prefetch 路径，所以本 PR 统一记录为
phase: "refined"，保持现有行为，并为后续 Fast/Refined 拆分预留字段。本 PR 不改变 recall selection 行为，也不实现 Fast/Refined 投递逻辑。为什么需要
Issue #7040 需要可靠观察 auto-memory recall 是否真正交付。现在只能看到 recall 选中了 memory，但看不到它是否进入主模型 prompt、是否晚到后在安全 continuation 点投递，或者是否因为 turn 结束而被丢弃。缺少这个信号，就很难评估 first-turn delivery、late delivery 和 cancellation 行为。
这个 PR 补上 delivery/discard 信号，并且只记录低基数字段。事件不会包含 query 文本、query hash、memory 内容、文件路径、project path、session/message id、raw error 或 secret。
Reviewer Test Plan
如何验证
运行聚焦的 telemetry 和 client 测试。确认 initial prompt 注入、ToolResult 注入、空 recall 结果、reset 取消、新 query 顶替都会产生 delivery telemetry，并确认 metrics 只携带低基数字段。
E2E 可以用 tmux 启动 CLI，设置
QWEN_CODE_MEMORY_LOCAL=1并开启 telemetry outfile。准备一个本地.qwen/memory/文件，包含ultraviolet parsnip这类唯一关键词，然后询问相关 memory。检查 telemetry outfile 中是否有qwen-code.memory.recall.delivery、qwen-code.memory.recall.delivery.count和qwen-code.memory.recall.delivery.latency。确认每条 delivery event 只有一个终态，并且不包含 query、memory 内容、路径、session/message id、raw error 或 secret。证据 Before & After
Before：auto-memory recall telemetry 只记录 selection 结果，无法区分 selected memory 是已投递、晚投递，还是最终被丢弃。
After：在
/tmp/pr1-memory-test的 tmux E2E 中观察到了qwen-code.memory.recall.delivery事件和 delivery metrics。已验证正常 recall、ToolResult delivery 和 abort discard。new_query和reset在真实 TUI 中受时序限制，recall 完成太快，未稳定复现，但对应取消路径已有单测覆盖。Tested on
Environment
macOS 本地 worktree，构建 bundle 后运行聚焦 Vitest 测试，并用 tmux 进行 CLI E2E，telemetry 通过 outfile 落盘。
Risk & Scope
new_query和resetdiscard reason，因为 recall 完成太快，但单测已覆盖这些路径。Linked Issues
Refs #7040