diff --git a/docs/design/2026-05-15-async-memory-recall-design.md b/docs/design/2026-05-15-async-memory-recall-design.md index f11b2ac5d23..d064eeee7be 100644 --- a/docs/design/2026-05-15-async-memory-recall-design.md +++ b/docs/design/2026-05-15-async-memory-recall-design.md @@ -5,6 +5,14 @@ **Related issues:** #3761, #3759 **Related PRs:** #3814, #3866 +> **Updated 2026-08-08:** The UserQuery consume point now waits for at most +> 100 ms. If the recall settles inside that budget it is delivered initially; +> if it does not, a deterministic fast result is delivered instead of nothing, +> and the pending model-selected result is preserved for ToolResult delivery +> with the fast-delivered documents excluded. The zero-wait UserQuery +> statements and behavior-table rows below describe the original design and are +> superseded by `2026-08-08-native-memory-recall-reliability.md`. + --- ## Problem diff --git a/docs/design/2026-08-08-native-memory-recall-reliability.md b/docs/design/2026-08-08-native-memory-recall-reliability.md new file mode 100644 index 00000000000..ba5763e57b8 --- /dev/null +++ b/docs/design/2026-08-08-native-memory-recall-reliability.md @@ -0,0 +1,271 @@ +# Native Memory Recall Reliability + +## Problem + +Managed-memory recall starts asynchronously for each user query. The initial +request originally performed a zero-wait consume, so a useful selector result +could miss the first prompt. If the turn has no tool call, that result has no +later safe delivery point and is discarded. + +A fixed 100 ms initial budget was the first attempt at a fix. Measurement +showed it is not sufficient on its own. Recall awaits the model selector +whenever a `Config` is present — the normal case — and that selector is a +network side query whose abort ceiling is 30 s. The budget is therefore +dominated by round-trip time, not by the incidental scheduler timing it was +sized for, so it expires on the common path. Delivery then falls through to the +ToolResult point, which a tool-free turn never reaches. Tool-free turns are +exactly the ones where user-level memory matters most: short questions answered +from context rather than from the repository. + +The model selector remains the normal precision gate. Its failure fallback had +two independent correctness problems: it tokenized only ASCII text and gave +every non-empty document a positive score even without a lexical match. + +## Decision + +Keep a single recall lifecycle and model-primary selection. Add one +deterministic delivery stage in front of it — not the two-stage shared-scan +Fast/Refined architecture originally proposed in RFC #7040. + +- Give user-query recall a 100 ms initial wait **ceiling**, not a fixed cost. + The wait ends on whichever comes first: recall settling, the deterministic + result being published, cancellation, or the ceiling. +- Deliver a result that settles inside the budget in the initial prompt. +- If the budget expires and the deterministic candidate pass found relevant + documents, deliver that bounded result instead of nothing. + `selectModelCandidateDocuments` already computes lexically ranked candidates + in order to build the model manifest, so the fast result reuses them and + costs no extra scan or I/O. It is capped at two documents + (`MAX_FAST_RECALL_DOCS`), well below the five-document prompt limit, because + it carries no model judgement. +- Leave recall pending after a fast delivery so the model-selected result still + lands at the existing same-query ToolResult delivery point. +- Exclude documents the fast phase already delivered from that later delivery, + rebuilding the prompt from what remains. Both results come from one scan, so + the selector never saw the fast documents as excluded and can legitimately + re-select them. When every selected document was already delivered, record + `already_delivered`; when the selector returned no documents at all, record + `no_relevant_results`. +- Do not abort recall merely because the initial budget expires. +- Preserve the existing cancellation and exactly-once terminal telemetry paths. + A cancelled turn delivers no fast result. + +The 100 ms budget stays internal, per RFC #7040's direction of a small fixed +internal budget determined by benchmark rather than exposed as public +configuration; telemetry can show whether a later change is justified. + +### The budget is a ceiling because the scan, not the selector, decides + +The fast result is published once recall has enumerated, read, and parsed the +memory tree — and this design removed the 200-document cap for recall, so that +scan grows with the tree. `recall-scan-latency.test.ts` measures the wall-clock +time from the recall call to that publication against a real temporary tree: + +| topics | median | share of the 100 ms budget | +| ------ | ------- | -------------------------- | +| 200 | ~29 ms | ~29% | +| 500 | ~70 ms | ~70% | +| 1000 | ~130 ms | ~130% | + +Two conclusions follow, and neither is visible in the deterministic _scoring_ +cost, which is microseconds. + +First, for any tree small enough to scan in time — which is the ordinary case, +where a user holds tens of topics rather than hundreds — the fast result is in +hand long before the ceiling. Spending the remainder waits for a model selector +that this design already assumes will miss the budget, so it is close to pure +added latency on every user turn. The wait therefore ends on the fast result. + +This has a consequence the code does not make obvious, so state it directly: +**on the initial turn, once the deterministic scorer matches anything, the fast +result is what gets delivered — regardless of how fast the selector is.** +`onFastResult` is published before recall issues the selector request at all, +so the recall promise is necessarily unsettled when the wait ends on it, and +the "prefer a settled recall" branch is reached only when no fast result +exists: no `Config`, or nothing matched lexically. + +That is the intended trade, not an oversight. A model side query does not +complete inside a 100 ms ceiling in production, so arbitrating between the two +would spend the remainder of the budget on every turn to win a race that does +not occur. Local verification against a loopback selector settling in 15 ms +confirms the behaviour and its bound: the fast result is delivered and the +model's picks are discarded — while the pre-change build delivered nothing at +all on that same turn. The selector's judgement still reaches the model, at the +ToolResult delivery point, with the fast documents excluded. + +Second, on a slow enough machine the scan alone exceeds the ceiling. The table +above is the conservative measurement; an independent run on faster hardware +recorded 9 ms / 21 ms / 46 ms for the same three sizes, all inside the ceiling. +The crossover is therefore a property of the machine, not a fixed topic count — +somewhere between roughly one thousand topics and never, depending on I/O +speed. Past it, a turn spends the whole budget and still delivers nothing, +which is worse than the zero-wait behaviour this design replaced. Ending the wait early does not fix +that case; it bounds it and removes the cost everywhere else. A persistent +catalog is the actual fix and remains out of scope, per +`2026-08-09-bounded-memory-recall-candidates.md`. + +### `MAX_RELEVANT_DOCS` is per delivery, not per turn + +`MAX_RELEVANT_DOCS = 5` bounds one prompt. It does not bound a turn. A turn +that fast-delivers two documents and then, at ToolResult, delivers five +documents the fast phase did not include puts **seven** documents in front of +the model. Deduplication removes repeats, not the sum. + +This is a deliberate consequence of dropping combined fast/refined budget +accounting, which RFC #7040 originally specified as a fill-to-five limit +across both phases. Keeping the combined limit means carrying a cross-phase +document budget through the delivery path — the same bookkeeping this design +declined for the duplicate-injection risk it introduces. Both prompts stay +individually bounded, each document body is still truncated to +`MAX_DOC_BODY_CHARS`, and the fast phase is capped at two, so the worst case +is bounded and small; it is simply not five. + +Should the aggregate ever need a hard ceiling, the cheap version is to pass +`limit - fastDeliveredPaths.size` as the refined limit rather than to +reintroduce a second budget. + +### Why not the original Fast/Refined architecture + +RFC #7040 specified two results produced from one shared scan, with the refined +pass excluding already-delivered fast documents and filling up to a combined +five-document limit. The delivery guarantee that design existed to provide is +worth having; its machinery is not. A second selection pathway needs its own +scan plumbing, its own budget accounting, and cross-phase document bookkeeping — +and that bookkeeping is the source of the duplicate-injection class of bug the +RFC itself warned about. Reusing the candidates the selector was already going +to score gets the same guarantee from one added callback and one exclusion set. + +### Telemetry: `phase` and `strategy` are orthogonal + +`phase` is the **delivery stage**: `fast` for a deterministic result injected +at budget expiry, `refined` for the model-selected result. `strategy` is the +**selection method**: `none`, `heuristic`, or `model`. They are not redundant +and neither subsumes the other. A `fast` delivery is always `heuristic`, but a +`refined` delivery is `model` normally and `heuristic` when the selector failed +and the fallback ran. Reading delivery-stage behaviour off `strategy` alone +would silently merge "the deterministic result arrived first" with "the model +selector broke". + +Improve the deterministic scorer, which now serves both the fast path and the +selector-failure fallback: + +- normalize query and document text with Unicode NFKC; +- keep runs of at least three non-CJK letters, marks, and digits as whole + tokens. `\p{L}`-based rather than `[a-z0-9]`, so Cyrillic, Greek, Arabic, + and accented Latin produce tokens instead of none. CJK is excluded per + character rather than by alternation order, because `\p{L}` also matches + Han and a Latin-initial run would otherwise swallow the CJK after it and + turn `abc漢字` into a single token; +- generate Unicode code-point bigrams for Han, Hiragana, Katakana, and Hangul + runs; +- ignore isolated CJK characters; +- bound fallback query tokens while retaining tokens from both ends; +- score only the body window that can be surfaced in the prompt; +- require a title, description, or body lexical match before applying a type + boost; +- weight each title and description token match above a body token match; +- break score ties by recency, then by input order, never by document type. + An alphabetical type comparison orders `feedback` before `project` before + `reference` before `user`, and `MAX_FAST_RECALL_DOCS` takes only the top + two, so a type tie-break would systematically drop user-level memory from + the fast result — the exact case the fast path exists to serve. Input order + as the final key keeps the project-before-user precedence, because recall + concatenates project documents ahead of user ones. + +## Non-goals + +- No second scan, second selector, or separate fast/refined budget accounting. +- No public recall timing or retrieval-mode setting. +- No new tokenizer or retrieval dependency. +- No change to memory writes, scopes, extraction, DREAM, forget, or compaction. +- No removal of the shared scanner's 200-document cap for non-recall callers. + Recall alone uses the bounded broad-candidate design documented in + `2026-08-09-bounded-memory-recall-candidates.md`. + +## Verification + +Recall quality is measured in `packages/core/src/memory/recall-eval.test.ts` +against a 51-case, 25-document labeled corpus, scored both by the shipped +scorer and by a frozen copy of the pre-change one so "no regression" is +reproducible rather than asserted. Delivery is measured separately in +`recall-delivery-eval.test.ts`, because a correct selection that never reaches +the model is worth nothing, and scan latency in `recall-scan-latency.test.ts`, +because a correct selection that is not ready in time reaches nothing either. + +The eval prints the corpus size and the Recall@5 a query-blind random scorer +would achieve on it (5 of 25 documents, so 20%), and a test keeps that floor +at or below 25% with the measured result well clear of it. A small corpus +flatters every design; the floor is what makes the headline readable. + +- Recall settling inside the budget is delivered initially. +- A budget expiry with deterministic candidates delivers that bounded result + rather than nothing, and leaves recall alive for later ToolResult delivery. +- The later delivery never repeats a document the fast phase already sent. +- Cancellation ends the bounded wait and prevents stale delivery. +- A fast result never crosses a query boundary. +- No-result queries stay silent under both designs. +- A labeled set covers Chinese, English, Japanese, Korean, mixed text, + NFKC normalization, body-only matches, no-result queries, answerable + queries that share no token with their document, and alphabetic scripts + outside ASCII and CJK (Cyrillic, Greek, accented Latin). +- The fast result is published inside the initial ceiling for tree sizes a + user can plausibly reach, measured against a real temporary memory tree + rather than modelled. +- The initial wait ends as soon as the deterministic result is published and + does not run to the ceiling; a wait with nothing to deliver still runs to + the ceiling and then proceeds without memory. +- The active-tool alias set is derived once per recall rather than once per + scanned document. +- Score ties are broken by recency rather than by document type, so a + user-typed document is not pushed out of the two-document fast result by a + tied feedback, project, or reference document. +- A result whose every document the fast phase already delivered is recorded + as `already_delivered` wherever it is discarded, not only at the ToolResult + consume point. A tool-free turn that delivered everything must not be + counted in the `no_safe_delivery_point` bucket; a partial overlap still is, + because the documents outside the fast set genuinely had no delivery point. +- Long CJK queries keep bounded scoring work and preserve both query ends. +- Existing active-tool noise filtering remains unchanged on the deterministic + candidate path. The model-selector failure fallback still triggers and + returns at most five documents; its scoring quality intentionally improves + per the scorer changes above (measured by the frozen-scorer comparison in + `recall-eval.test.ts`). + +### Known limitations + +- The delivery evaluation models selector latency rather than measuring it; a + network round trip cannot be timed in a unit test. Results are reported per + latency scenario, and the structural claim — that a selector slower than the + budget leaves a tool-free turn with no delivery point under the single-path + design — holds for every scenario above the budget. +- The fast result has no model judgement behind it. It is capped at two + documents to bound the cost of being wrong, but on a tool-free turn where the + selector never lands, a mis-ranked fast document is what the model sees. +- Scoring is substring-based, so a query token can match inside a longer word + ("owner" inside "ownership"). The evaluation corpus records one such case + rather than hiding it. +- The fast path closes the timing gap, not the matching gap. A query that + shares no token with its document produces no deterministic result, so a + tool-free turn asking it still ends with nothing delivered; only the model + selector can serve those, and on a tool-free turn it never lands. The + `semantic-no-lexical` slice of the corpus measures this directly — the + shipped scorer and the frozen pre-change scorer both score 0% Recall@5 on + it, so requiring a lexical match did not create the gap, but it does keep + the fast path silent there. This is why the headline tool-free delivery + figure is 92.3% and not 100%: the residual 7.7% is exactly that slice. +- On slow enough I/O the memory-tree scan exceeds the initial ceiling, and the + turn then spends the whole budget and still delivers nothing. The crossover + is machine-dependent: roughly a thousand topics on the slower of the two + machines measured, and not reached at all on the faster one. Ending the wait + on the fast result bounds this rather than removing it; the real fix is a + persistent catalog, which is out of scope. +- On the initial turn the model selector's judgement is not used when the + deterministic scorer matched, whatever the selector's latency. See the + ceiling section above; it reaches the model at ToolResult instead. +- Scripts written without word separators and outside the CJK set — Thai, + Khmer, Lao — now produce a token where they previously produced none, but + the token is the whole run. That is not segmentation, and such a query will + usually still match nothing. +- Recall can see older documents outside the shared 200-document scanner cap, + but non-recall callers, including Forget, keep the existing capped scanner. + A broader manageability pass is separate from this recall-only change. diff --git a/docs/design/2026-08-09-bounded-memory-recall-candidates.md b/docs/design/2026-08-09-bounded-memory-recall-candidates.md new file mode 100644 index 00000000000..236baed79f8 --- /dev/null +++ b/docs/design/2026-08-09-bounded-memory-recall-candidates.md @@ -0,0 +1,116 @@ +# Bounded Memory Recall Candidates + +## Problem + +The project and user memory scanners enumerate, read, and parse every topic, +then return only the 200 most recent documents. Recall uses those shared scanner +APIs, so an older relevant document outside either 200-document window cannot +reach the heuristic or model selector even though the expensive scan work has +already happened. The truncation key is recency, applied per scope and before +anything has looked at the query. + +The same capped APIs are also used by Forget, Indexer, Status, and Extraction. +Removing their limit globally would widen unrelated behavior. + +## Decision + +Keep the existing scanner APIs and their 200-document limit unchanged. Add +explicit all-topic variants used only by recall. + +Recall ranks the combined project and user pool before model selection: + +- retain up to 180 documents with a lexical match using the existing scorer; +- fill the remaining candidate slots by recency, preserving at least 20 recent + opportunities when enough lexical matches exist; +- interleave recent opportunities with lexical candidates so the manifest byte + budget cannot systematically exclude the entire recent reserve; +- send at most 200 candidates to the model selector; +- append manifest entries only while their cumulative UTF-8 size remains at or + below 25,000 bytes; +- validate selector output only against documents actually present in that + bounded manifest. + +The heuristic fallback continues to score the complete recall pool and still +returns at most five documents. Existing body and prompt limits remain +unchanged. + +### This is a change of truncation key, not a lifted ceiling + +"Removes the 200-document cap" is the wrong summary, and reviewers should +read the effect per pool size rather than as a uniform widening. What the +change actually does is replace a per-scope, query-blind recency truncation +with a global, query-aware one: + +- **Pool at or under 200 documents.** No document is excluded by count under + either design, but the 25,000-byte manifest budget is a ceiling the old path + did not have, and it binds far earlier than the document count suggests + (see below). The interleaving above exists so that truncation cannot fall + entirely on the recent reserve. +- **Pool between 200 and 400 documents, neither scope over 200.** The old + path sent every document to the selector — up to 200 project plus 200 user. + The new path sends at most 200, and in practice the byte budget cuts it + further: a measured run with 150 project plus 150 user documents sent 300 + manifest lines under the old path and **94** under the new one. The + candidates that survive are chosen by lexical relevance plus a recency + reserve rather than by recency alone, which is the intended trade, but the + reduction is larger than the document cap implies. +- **Either scope over 200 documents.** This is the case the change is for. + An old, lexically matching document that the recency cap made permanently + invisible can now be selected. Measured: with 251 documents in one scope and + the only lexical match the oldest, the old path produced a 200-line manifest + without the target; the new path produced a 96-line manifest with the target + first. + +### The binding constraint is `MAX_MODEL_MANIFEST_BYTES`, not the document cap + +`MAX_MODEL_CANDIDATE_DOCS = 200` reads like the limit but rarely is one. Each +manifest line carries an absolute file path and an ISO-8601 timestamp before +the description, so its fixed overhead is on the order of 150–250 bytes for an +ordinary project path. Against a 25,000-byte budget that binds somewhere around +90–150 documents, which is why both measurements above land in the nineties +rather than at 200. + +Two things follow. Deployments should read the byte budget, not the document +cap, as the real candidate ceiling. And the recency reserve only survives +truncation because it is interleaved with the lexical candidates rather than +appended after them — at a cut in the nineties, an appended reserve would be +discarded in full. + +The manifest byte budget also packs rather than prefixes: a document whose +line does not fit is skipped and later, shorter lines are still considered. +A long-description document can therefore be dropped while a lower-ranked one +is kept. + +Forget, Indexer, Status, and Extraction keep the capped scanner. That preserves +their current behavior but means an older document can become recallable before +it becomes manageable by those non-recall flows. + +## Failure and compatibility boundaries + +Project scanning remains required. User scanning remains best-effort. Invalid +or unreadable files keep the existing skip behavior. Empty candidate manifests +return no model selection rather than sending an unbounded request. + +There is no public setting, persistent index, new dependency, provider API, or +second selection pathway. Each recall enumerates, reads, and parses the full +project and user memory trees once, then performs O(n) local ranking and +active-tool filtering over the parsed documents. The deterministic fast path +described in `2026-08-08-native-memory-recall-reliability.md` reuses the +candidates produced by that single pass, so it adds no scan, no ranking work, +and no state machine — only an earlier delivery point for results already +computed. The model candidate count and manifest are +bounded, but the local I/O and filtering work grow with the memory tree; a +persistent catalog requires separate measurement and evidence. + +## Verification + +- A deliberately old relevant topic beyond the regular 200-document result is + recalled from a real temporary memory tree. +- The regular scanner still returns 200 documents and omits that topic. +- The model candidate set contains the lexical target and recent reserve while + remaining at 200 documents. +- A manifest built from large multibyte descriptions stays within 25,000 UTF-8 + bytes. +- A real temporary memory-tree integration test verifies overflow-topic recall; + client tests independently verify bounded initial waiting and later + ToolResult delivery. diff --git a/docs/design/auto-memory/memory-system.md b/docs/design/auto-memory/memory-system.md index b8b81c38fa7..6d35ce14f0f 100644 --- a/docs/design/auto-memory/memory-system.md +++ b/docs/design/auto-memory/memory-system.md @@ -350,35 +350,58 @@ flowchart TD ```mermaid flowchart TD - A[resolveRelevantAutoMemoryPromptForQuery] --> B[scanAutoMemoryTopicDocuments\n扫描所有主题文件] - B --> C[filterExcludedAutoMemoryDocuments\n过滤本轮已写入的文件] + A[resolveRelevantAutoMemoryPromptForQuery] --> B[scanAllAutoMemoryTopicDocuments +\nscanAllUserAutoMemoryTopicDocuments\n扫描项目级与用户级全部主题文件] + B --> C[filterExcludedAutoMemoryDocuments\n合并作用域并过滤排除列表中的文件] C --> D{query 为空\n或 docs 为空\n或 limit <= 0?} D -- 是 --> E[返回空 prompt\nstrategy: none] D -- 否 --> F{是否配置了 Config?} - F -- 是 --> G[selectRelevantAutoMemoryDocumentsByModel\n发起 side query 请求模型选择] - G --> H{模型返回结果?} - H -- 有文档 --> I[strategy: model] - H -- 无文档 --> J[strategy: none\n仍然返回空] - G -- "失败/异常" --> K[回退到启发式选择] - F -- 否 --> K - K --> L[tokenize query\n提取 ≥3 字符的 token] - L --> M[scoreDocument 打分\n关键词匹配 +2 / 类型关键词 +1 / 有内容 +1] - M --> N[过滤 score=0 的文档\n按分数降序排列，取 Top 5] - N --> O{有得分文档?} - O -- 是 --> P[strategy: heuristic] - O -- 否 --> J - I --> Q[buildRelevantAutoMemoryPrompt\n构建 Relevant Memory 区块] - P --> Q - Q --> R[返回注入主系统提示的 prompt 片段] + F -- 是 --> G[selectModelCandidateDocuments\n词法候选 + recent reserve\n最多 200 篇且交错排列] + G --> H[selectRelevantAutoMemoryDocumentsByModel\n构建最多 25 KB manifest\n发起 side query 请求模型选择] + H --> I{模型返回结果?} + I -- 有文档 --> J[strategy: model] + I -- 无文档 --> K[strategy: none\n仍然返回空] + H -- "失败/异常" --> L[复用已计算的启发式排序] + F -- 否 --> M[tokenize query\nNFKC + 非 CJK 字母整串 + CJK bigram\n最多 64 个 token] + M --> N[scoreDocument 打分\ntitle +4 / description +3 / body +1\n词法命中后类型加成最多 +2] + N --> O[过滤 score=0 的文档\n按分数降序、mtime 降序、输入顺序排列\n取 Top 5] + L --> O + O --> P{有得分文档?} + P -- 是 --> Q[strategy: heuristic] + P -- 否 --> K + J --> R[buildRelevantAutoMemoryPrompt\n构建 Relevant Memory 区块] + Q --> R + R --> S[返回注入主系统提示的 prompt 片段] ``` +> **关于 200 上限：这是换了截断依据，不是抬高了天花板。** 旧路径按 scope 各自 +> 保留最近 200 篇，截断发生在看 query 之前；新路径先扫全量，再按词法相关性 + +> recency reserve 选出最多 200 篇候选。因此效果分三档：总量 ≤200 时两者都不按 +> 数量丢弃（但新增了 25 KB manifest 上限，长 description 场景可能被截）；总量 +> 在 200–400 且单个 scope 不超 200 时，旧路径会把全部（最多 400 篇）送给 +> Selector，新路径最多 200 篇，且实测中 25 KB 的 manifest 预算会先于数量上限 +> 生效——150+150 篇的实测里旧路径送 300 行、新路径只送 94 行，**候选变少的幅度 +> 比数量上限暗示的更大**；只有单个 scope 超过 200 时， +> 才是这次真正要解决的场景——旧的 recency 上限会让老而相关的文档永久不可见。 +> 详见 `docs/design/2026-08-09-bounded-memory-recall-candidates.md`。 + **评分规则（启发式）**： -| 条件 | 加分 | -| -------------------------------- | ---------------- | -| query token 出现在文档内容中 | +2（每个 token） | -| query token 是该类型的特征关键词 | +1（每个 token） | -| 文档 body 非空 | +1 | +| 条件 | 加分 | +| ------------------------------------------ | ------------------- | +| query token 出现在 title | +4（每个 token） | +| query token 出现在 description | +3（每个 token） | +| query token 出现在 body 前 1200 字符 | +1（每个 token） | +| 至少一次词法命中后，token 是类型特征关键词 | +1，整篇文档最多 +2 | + +> **Tokenize 规则**：NFKC 归一化并转小写后，Han/Hiragana/Katakana/Hangul 连续片段 +> 按 code point bigram 切分（单字不产生 token）；其余至少 3 个字母、组合符或数字的 +> 连续片段整串保留。后者基于 `\p{L}` 而非 `[a-z0-9]`，因此西里尔、希腊、阿拉伯和 +> 带重音拉丁文都能产生 token。CJK 是**逐字符**排除的，不能只依赖正则分支顺序—— +> `\p{L}` 也匹配 Han，否则 `abc漢字` 会被并成一个 token。Thai/Khmer/Lao 这类 +> 无分词符又不在 CJK 集合内的文字，会整段变成一个 token：比之前完全没有 token 强， +> 但不是分词。 +> +> **同分排序**：按 mtime 降序，再按输入顺序（稳定排序），**不按 type**。 **每种类型的特征关键词**： @@ -389,11 +412,77 @@ flowchart TD **Prompt 构建规则**： -- 最多注入 5 篇文档（`MAX_RELEVANT_DOCS`） +- 单次注入最多 5 篇文档（`MAX_RELEVANT_DOCS`） - 每篇文档 body 截断至 1200 字符（`MAX_DOC_BODY_CHARS`） - 超出截断时追加提示："NOTE: Relevant memory truncated for prompt budget." - 包含文档的新鲜度信息（基于文件 mtime） +> **`MAX_RELEVANT_DOCS` 限制的是单次注入，不是单轮总量。** Fast 阶段投递 2 篇、 +> ToolResult 阶段又投递 5 篇全新文档时，本轮进入模型的是 **7 篇**——去重只消除 +> 重复，不压缩总和。这是放弃跨阶段预算核算的有意结果（见 +> `2026-08-08-native-memory-recall-reliability.md`）：两次 Prompt 各自有界， +> 每篇 body 仍截断到 1200 字符，Fast 上限为 2，因此最坏情况有界且不大，只是不等于 5。 + +### 投递时机（Delivery） + +"选中了 Memory" 不等于 "主模型看到了 Memory"。Recall 在 UserQuery 到达时异步启动， +投递发生在两个时机： + +```mermaid +flowchart TD + A[UserQuery 到达\n启动 Recall Prefetch] --> B{等待结束\n以先到者为准:\nRecall 完成 / Fast 就绪 /\n取消 / 100 ms 上限} + B --> B1{Recall 是否完成?} + B1 -- 是 --> C{选中结果非空?} + C -- 是 --> C1[注入首轮 Prompt\nphase: refined] + C -- 否 --> C0[丢弃\nno_relevant_results] + B1 -- 否 --> D{是否有确定性\nFast 结果?} + D -- 是 --> E[注入首轮 Prompt\nphase: fast\n最多 2 篇 MAX_FAST_RECALL_DOCS] + D -- 否 --> F[首轮不注入] + E --> G[Recall 继续运行] + F --> G + G --> H{本轮是否有\nToolResult?} + H -- 是 --> I{Recall 是否已完成?} + I -- 是 --> J[排除 Fast 已投递文档\n按剩余文档重建 Prompt] + I -- 否 --> M + J --> J1{还有剩余文档?} + J1 -- 是 --> K[注入 ToolResult\nphase: refined] + J1 -- 否 --> L{Recall 选中了文档?} + L -- 是 --> L1[丢弃\nalready_delivered] + L -- 否 --> L2[丢弃\nno_relevant_results] + H -- 否 --> M{选中文档是否\n全部已被 Fast 投递?} + M -- 是 --> L1 + M -- 否 --> M1[丢弃\nno_safe_delivery_point] +``` + +**为什么需要 Fast 阶段**：当存在 Config 时 Recall 会等待 Model Selector， +而它是一次网络 Side Query（中止上限 30 秒），因此 100 ms 预算通常会超时。 +若没有 Fast 阶段，**没有工具调用的轮次将完全拿不到 Memory**——而这正是 +用户级 Memory 最重要的场景。Fast 结果复用 `selectModelCandidateDocuments` +为 Model Manifest 已经算好的候选，不产生额外扫描或 I/O。 + +**100 ms 是上限而不是固定开销**：Fast 结果在 Recall 扫完 Memory 树之后才发布， +所以真正决定它能否赶上的是**扫描耗时**，不是打分耗时（后者是微秒级）。 +`recall-scan-latency.test.ts` 在真实临时 Memory 树上实测：200 篇约 29 ms、 +500 篇约 70 ms、1000 篇约 130 ms。对能在预算内扫完的树（普通用户的常见情况）， +Fast 就绪后继续等待只是在等一个本设计已经假定赶不上的 Model Selector， +因此等待会在 Fast 就绪时立即结束。超过约 1000 篇时扫描本身就超预算， +该轮会付满 100 ms 且什么都投不到——提前结束等待只能把这种情况**限制住**， +消除不了它。 + +**Fast 阶段的边界**：Fast 结果就是确定性结果，因此它只能解决**时机**问题， +解决不了**匹配**问题。与文档没有任何词面重叠的 Query 产生不了 Fast 结果， +这类 Query 在无工具回合仍然拿不到 Memory——只有 Model Selector 能覆盖它们， +而无工具回合等不到 Selector。语料中的 `semantic-no-lexical` 分片专门测量这一点。 + +**去重**：两个阶段来自同一次扫描，Model Selector 并未把 Fast 文档视为已排除， +因此 ToolResult 投递前必须过滤掉 Fast 已投递的 `filePath` 并重建 Prompt。 + +**丢弃口径**：同一条规则也适用于取消路径。若最终选中的文档已被 Fast 阶段 +全部投递，无论本轮是因为无工具调用、New Query、Reset、Abort 还是 Shutdown +结束，都记为 `already_delivered` 而不是对应的取消原因——否则「Memory 从未 +到达模型」这一桶会被实际已送达的回合灌水。只有部分重叠时仍记取消原因， +因为不在 Fast 集合里的那些文档确实没有投递点。 + --- ## Forget — 遗忘 @@ -489,6 +578,25 @@ flowchart TD | `strategy` | `'none'` \| `'heuristic'` \| `'model'` | 选择策略 | | `duration_ms` | number | 总耗时（毫秒） | +### Recall Delivery 遥测 + +记录选中的 Memory 是否真的送达主模型（Selection 事件无法回答这个问题）。 + +| 字段 | 类型 | 说明 | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | +| `phase` | `'fast'` \| `'refined'` | **投递阶段** | +| `delivery_point` | `'initial'` \| `'tool_result'` \| `'discarded'` | 投递位置 | +| `discard_reason` | `'no_safe_delivery_point'` \| `'new_query'` \| `'reset'` \| `'abort'` \| `'shutdown'` \| `'no_relevant_results'` \| `'already_delivered'` | 丢弃原因 | +| `strategy` | `'none'` \| `'heuristic'` \| `'model'` | **选择方式** | +| `docs_selected` | number | 结果文档数（投递事件为实际投递数；discarded 事件为选中数） | +| `latency_ms` | number | 自发起的耗时 | + +> **`phase` 与 `strategy` 正交，互不替代。** `phase` 描述**何时**送达：`fast` 是预算 +> 超时后注入的确定性结果，`refined` 是 Model Selector 选出的结果。`strategy` 描述 +> **如何**选出。`fast` 投递必然是 `heuristic`；`refined` 投递常规为 `model`， +> 在 Selector 失败走 Fallback 时为 `heuristic`。仅凭 `strategy` 判断阶段， +> 会把"确定性结果先到"与"Selector 故障"混为一谈。 + --- ## 相关源文件索引 diff --git a/packages/core/src/core/client.test.ts b/packages/core/src/core/client.test.ts index fdf3391af8b..283fb9a64d4 100644 --- a/packages/core/src/core/client.test.ts +++ b/packages/core/src/core/client.test.ts @@ -5095,6 +5095,664 @@ hello expect(requestText).not.toContain(''); }); + // Delivery-stage coverage for the deterministic fast path. The model + // selector is a network side query, so on a turn that makes no tool call + // the refined result has no safe delivery point at all. These cases pin + // the fast path that closes that gap, plus the dedupe, cancellation, and + // exactly-once guarantees it must not break. + const fastDoc = (filePath: string, body: string) => ({ + type: 'user' as const, + filePath, + relativePath: filePath.split('/').at(-1)!, + filename: filePath.split('/').at(-1)!, + title: 'User Memory', + description: 'User preferences', + body, + mtimeMs: 1, + }); + + const toolCallStream = () => + (async function* () { + yield { type: 'content', value: 'Hello' }; + yield { + type: 'tool_call_request', + value: { + callId: 'call-1', + name: 'foo', + args: {}, + isClientInitiated: false, + prompt_id: 'prompt-id-fast', + }, + }; + })(); + + it('delivers the deterministic fast result on a tool-free turn when the selector is still in flight', async () => { + vi.useFakeTimers(); + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + // Selector never settles — stands in for a slow round trip. + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-tool-free', + ), + ); + + // The deterministic result was already published, so the budget has + // nothing left to wait for and the request goes out without spending it. + await vi.advanceTimersByTimeAsync(0); + await done; + + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + }); + + it('ends the initial wait as soon as the deterministic result arrives', async () => { + vi.useFakeTimers(); + // Stands in for the memory-tree scan: the fast result is not ready when + // the wait begins, but lands well before the budget expires. + const SCAN_MS = 30; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, SCAN_MS); + // Selector never settles — stands in for a slow round trip. + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-early-return', + ), + ); + + await vi.advanceTimersByTimeAsync(SCAN_MS - 1); + expect(mockTurnRunFn).not.toHaveBeenCalled(); + // The remaining ~70 ms of budget is never spent. + await vi.advanceTimersByTimeAsync(1); + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + + await vi.advanceTimersByTimeAsync(100); + await done; + }); + + /** + * Pins the consequence of ending the wait on the fast result, which local + * verification on a real stack surfaced as broader than "slow selectors": + * once the deterministic scorer matches, the initial turn delivers the + * fast result whatever the selector's latency. + * + * `onFastResult` is published before recall issues the selector request, + * so the recall promise cannot be settled when the wait ends on it. This + * is the intended trade — a model side query does not return inside the + * ceiling in production, so arbitrating would cost every turn the rest of + * the budget to win a race that does not happen — and the selector's + * judgement still lands at ToolResult. Recorded as a decision so a future + * reader does not mistake it for an accident. + */ + it('delivers the fast result even when the selector settles inside the budget', async () => { + vi.useFakeTimers(); + const SCAN_MS = 10; + const SELECTOR_MS = 15; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, SCAN_MS); + // Settles comfortably inside the 100 ms ceiling — and still loses. + return new Promise((resolve) => { + setTimeout( + () => + resolve({ + prompt: '## Relevant memory\n\nRefined model result.', + selectedDocs: [fastDoc('/m/refined.md', '- refined')], + strategy: 'model', + }), + SELECTOR_MS, + ); + }); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-beats-quick-selector', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(200); + await done; + + const initialRequest = mockTurnRunFn.mock.calls[0]?.[1] as unknown[]; + expect(initialRequest).toEqual( + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + ); + expect(initialRequest).not.toEqual( + expect.arrayContaining([ + expect.stringContaining('Refined model result.'), + ]), + ); + expect(logMemoryRecallDelivery).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + phase: 'fast', + delivery_point: 'initial', + strategy: 'heuristic', + }), + ); + }); + + it('still delivers the model-selected result at ToolResult after a fast initial delivery', async () => { + vi.useFakeTimers(); + let settleRecall: + | ((value: { + prompt: string; + selectedDocs: Array>; + strategy: 'model'; + }) => void) + | undefined; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + return new Promise((resolve) => { + settleRecall = resolve; + }); + }); + + mockTurnRunFn.mockReturnValue(toolCallStream()); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const userDone = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-then-refined', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + await userDone; + + expect(mockTurnRunFn).toHaveBeenLastCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + + // Selector lands between turns with a different document. + settleRecall!({ + prompt: '## Relevant memory\n\nRefined model result.', + selectedDocs: [fastDoc('/m/refined.md', '- refined')], + strategy: 'model', + }); + await vi.advanceTimersByTimeAsync(0); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'tool result turn' }; + })(), + ); + await fromAsync( + client.sendMessageStream( + [{ functionResponse: { name: 'foo', response: { ok: true } } }], + new AbortController().signal, + 'prompt-id-fast-then-refined-tool', + { type: SendMessageType.ToolResult }, + ), + ); + + expect(mockTurnRunFn).toHaveBeenLastCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Refined model result.'), + ]), + expect.any(AbortSignal), + ); + }); + + it('does not re-deliver a document the fast phase already injected', async () => { + vi.useFakeTimers(); + const overlapping = fastDoc('/m/overlap.md', '- overlapping'); + let settleRecall: + | ((value: { + prompt: string; + selectedDocs: Array>; + strategy: 'model'; + }) => void) + | undefined; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nOverlapping memory body.', + selectedDocs: [overlapping], + strategy: 'heuristic', + }); + return new Promise((resolve) => { + settleRecall = resolve; + }); + }); + + mockTurnRunFn.mockReturnValue(toolCallStream()); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const userDone = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-dedupe', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + await userDone; + + // The selector re-selects the fast document alongside a genuinely new + // one — it never saw the fast delivery, so overlap is expected. + // Markers live only in the selector's own prompt string. Dedupe must + // rebuild the prompt from the remaining documents, dropping them; if the + // result were passed through untouched the markers would survive. + settleRecall!({ + prompt: '## Relevant memory\n\nOVERLAP_MARKER\n\nNEW_MARKER', + selectedDocs: [overlapping, fastDoc('/m/new.md', '- brand new')], + strategy: 'model', + }); + await vi.advanceTimersByTimeAsync(0); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'tool result turn' }; + })(), + ); + await fromAsync( + client.sendMessageStream( + [{ functionResponse: { name: 'foo', response: { ok: true } } }], + new AbortController().signal, + 'prompt-id-fast-dedupe-tool', + { type: SendMessageType.ToolResult }, + ), + ); + + const toolRequest = mockTurnRunFn.mock.calls.at(-1)?.[1] as unknown[]; + const toolText = JSON.stringify(toolRequest); + // The genuinely new document still reaches the model, rendered from its + // own body by the rebuilt prompt. + expect(toolText).toContain('brand new'); + // The overlapping document was already in front of the model from the + // fast delivery; sending it again would duplicate context. Passing the + // selector result through unchanged would leave both markers intact. + expect(toolText).not.toContain('OVERLAP_MARKER'); + expect(toolText).not.toContain('- overlapping'); + }); + + it('logs already-delivered discards with the selector count', async () => { + vi.useFakeTimers(); + const overlapping = fastDoc('/m/overlap.md', '- overlapping'); + let settleRecall: + | ((value: { + prompt: string; + selectedDocs: Array>; + strategy: 'model'; + }) => void) + | undefined; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nOverlapping memory body.', + selectedDocs: [overlapping], + strategy: 'heuristic', + }); + return new Promise((resolve) => { + settleRecall = resolve; + }); + }); + + mockTurnRunFn.mockReturnValue(toolCallStream()); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const userDone = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-dedupe-discard', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + await userDone; + + settleRecall!({ + prompt: '## Relevant memory\n\nOVERLAP_MARKER', + selectedDocs: [overlapping], + strategy: 'model', + }); + await vi.advanceTimersByTimeAsync(0); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'tool result turn' }; + })(), + ); + await fromAsync( + client.sendMessageStream( + [{ functionResponse: { name: 'foo', response: { ok: true } } }], + new AbortController().signal, + 'prompt-id-fast-dedupe-discard-tool', + { type: SendMessageType.ToolResult }, + ), + ); + + expect(logMemoryRecallDelivery).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + phase: 'refined', + delivery_point: 'discarded', + strategy: 'model', + docs_selected: 1, + discard_reason: 'already_delivered', + }), + ); + }); + + /** + * Tool-free turn where the selector lands *after* the fast delivery but + * before the turn ends: the handle is discarded, so the reason it records + * is the only delivery signal this shape of turn produces. + */ + const runFastDiscardTurn = async ( + promptId: string, + fastDocs: Array>, + refinedDocs: Array>, + ) => { + let settleRecall: + | ((value: { + prompt: string; + selectedDocs: Array>; + strategy: 'model'; + }) => void) + | undefined; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: fastDocs, + strategy: 'heuristic', + }); + return new Promise((resolve) => { + settleRecall = resolve; + }); + }); + + // Held open so the selector can settle mid-turn; without it the turn + // ends first and the discard sees no result at all. + let releaseStream: (() => void) | undefined; + mockTurnRunFn.mockReturnValue( + (async function* () { + await new Promise((resolve) => { + releaseStream = resolve; + }); + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + promptId, + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + settleRecall!({ + prompt: '## Relevant memory\n\nREFINED_MARKER', + selectedDocs: refinedDocs, + strategy: 'model', + }); + await vi.advanceTimersByTimeAsync(0); + releaseStream!(); + await done; + }; + + it('reports a fully fast-delivered result as already-delivered, not as a lost one', async () => { + vi.useFakeTimers(); + const overlapping = fastDoc('/m/overlap.md', '- overlapping'); + await runFastDiscardTurn( + 'prompt-id-fast-discard-already-delivered', + [overlapping], + [overlapping], + ); + + expect(logMemoryRecallDelivery).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + phase: 'refined', + delivery_point: 'discarded', + discard_reason: 'already_delivered', + docs_selected: 1, + }), + ); + expect(logMemoryRecallDelivery).not.toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + discard_reason: 'no_safe_delivery_point', + }), + ); + }); + + it('still reports a partly fast-delivered result as having no safe delivery point', async () => { + vi.useFakeTimers(); + const overlapping = fastDoc('/m/overlap.md', '- overlapping'); + // `/m/extra.md` never reached the model, so the turn really did lose it. + const undelivered = fastDoc('/m/extra.md', '- extra'); + await runFastDiscardTurn( + 'prompt-id-fast-discard-partial', + [overlapping], + [overlapping, undelivered], + ); + + expect(logMemoryRecallDelivery).toHaveBeenCalledWith( + mockConfig, + expect.objectContaining({ + phase: 'refined', + delivery_point: 'discarded', + discard_reason: 'no_safe_delivery_point', + }), + ); + }); + + it('delivers no fast result when the turn is cancelled inside the initial window', async () => { + vi.useFakeTimers(); + const controller = new AbortController(); + // The fast result must still be in flight when the abort lands, + // otherwise the wait would already have ended on its arrival and there + // would be no window left to cancel inside. + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, 80); + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + controller.signal, + 'prompt-id-fast-cancelled', + ), + ).catch(() => {}); + + await vi.advanceTimersByTimeAsync(50); + controller.abort(); + await vi.advanceTimersByTimeAsync(100); + await done; + + expect(mockTurnRunFn).not.toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + }); + + it('does not leak a fast result across query boundaries', async () => { + vi.useFakeTimers(); + let call = 0; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + call += 1; + if (call === 1) { + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFirst turn fast result.', + selectedDocs: [fastDoc('/m/first.md', '- first')], + strategy: 'heuristic', + }); + } + // Neither recall settles; the second turn must not inherit the first + // turn's fast result. + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const first = fromAsync( + client.sendMessageStream( + [{ text: 'First question' }], + new AbortController().signal, + 'prompt-id-fast-leak-1', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + await first; + + mockTurnRunFn.mockClear(); + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello again' }; + })(), + ); + + const second = fromAsync( + client.sendMessageStream( + [{ text: 'Second question' }], + new AbortController().signal, + 'prompt-id-fast-leak-2', + { type: SendMessageType.UserQuery }, + ), + ); + await vi.advanceTimersByTimeAsync(100); + await second; + + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.not.arrayContaining([ + expect.stringContaining('First turn fast result.'), + ]), + expect.any(AbortSignal), + ); + }); + it('should prepend relevant managed auto-memory prompt when recall returns content', async () => { mockMemoryManager.recall.mockResolvedValue({ prompt: '## Relevant memory\n\nUser prefers terse responses.', @@ -5218,9 +5876,13 @@ hello ); }); - it('should not block the main request when auto-memory recall is slow', async () => { - // Recall never settles — settledAt stays null so the UserQuery consume - // point skips it and turn.run() is called immediately without memory. + it('should hold the main request for exactly the initial recall budget when recall never settles', async () => { + // Recall never settles and never publishes a deterministic result, so + // nothing can end the wait early. Fake timers pin the ceiling: the + // request must still be blocked 1 ms inside the budget and proceed, + // without memory, the moment the budget expires. This is also the shape + // of a memory tree whose scan is slower than the budget. + vi.useFakeTimers(); mockMemoryManager.recall.mockReturnValue(new Promise(() => {})); const mockStream = (async function* () { @@ -5234,16 +5896,23 @@ hello }; client['chat'] = mockChat as GeminiChat; - const stream = client.sendMessageStream( - [{ text: 'Quick question' }], - new AbortController().signal, - 'prompt-id-slow-memory', + const done = fromAsync( + client.sendMessageStream( + [{ text: 'Quick question' }], + new AbortController().signal, + 'prompt-id-slow-memory', + ), ); - for await (const _ of stream) { - // consume stream - } - // turn.run() must have been called without the slow memory + // Drain microtasks up to the consume point, then stop 1 ms short of + // the 100 ms budget: the request must still be held. + await vi.advanceTimersByTimeAsync(99); + expect(mockTurnRunFn).not.toHaveBeenCalled(); + + // Budget expiry: the request proceeds without the slow memory. + await vi.advanceTimersByTimeAsync(1); + await done; + expect(mockTurnRunFn).toHaveBeenCalledWith( 'test-model', expect.not.arrayContaining([ @@ -5253,6 +5922,61 @@ hello ); }); + it('should end the initial wait early when recall settles inside the budget', async () => { + // Fake timers pin the early-exit contract: once recall settles the + // request proceeds immediately with the memory — it must not run out + // the remaining budget. Dropping the settle listener in + // tryConsumeMemoryPrefetch would leave the request blocked until the + // full budget, which this assertion catches. + vi.useFakeTimers(); + mockMemoryManager.recall.mockImplementation( + () => + new Promise((resolve) => { + setTimeout( + () => + resolve({ + prompt: '## Relevant memory\n\nBounded memory result.', + selectedDocs: [], + strategy: 'model', + }), + 10, + ); + }), + ); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'Quick question' }], + new AbortController().signal, + 'prompt-id-bounded-memory', + ), + ); + + // Recall settles 10 ms in; the request must already be proceeding, + // 90 ms short of the budget. + await vi.advanceTimersByTimeAsync(10); + expect(mockTurnRunFn).toHaveBeenCalled(); + await done; + + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + '## Relevant memory\n\nBounded memory result.', + ]), + expect.any(AbortSignal), + ); + }); + it('should inject auto-memory at UserQuery consume point when recall already settled', async () => { // mockResolvedValue settles synchronously; by the time the consume-point // check runs (after at least one await), settledAt is set. @@ -5407,11 +6131,13 @@ hello strategy: 'model'; }) => void) | undefined; - mockMemoryManager.recall.mockReturnValue( - new Promise((resolve) => { + let recallSignal: AbortSignal | undefined; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + recallSignal = options.abortSignal; + return new Promise((resolve) => { resolveRecall = resolve; - }), - ); + }); + }); // The model requests a tool call so pendingToolCalls is non-empty and // the prefetch is preserved for the subsequent ToolResult turn. @@ -5454,6 +6180,7 @@ hello ]), expect.any(AbortSignal), ); + expect(recallSignal?.aborted).toBe(false); // Recall settles between turns resolveRecall!({ @@ -6172,6 +6899,189 @@ hello expect(abortHandlerInvoked).toBe(true); }); + it('should end the bounded initial wait when the prefetch is cancelled', async () => { + vi.useFakeTimers(); + const controller = new AbortController(); + const handle = { + promise: new Promise(() => {}), + settledAt: null, + result: null, + consumed: false, + terminalLogged: false, + fastResultRef: { current: null }, + fastDelivered: false, + fastDeliveredPaths: new Set(), + firedAt: Date.now(), + controller, + }; + client['pendingMemoryPrefetch'] = handle; + const privateClient = client as unknown as { + tryConsumeMemoryPrefetch: ( + deliveryPoint: 'initial', + waitMs: number, + ) => Promise; + cancelPendingMemoryPrefetch: (reason: 'abort') => void; + }; + + const consume = privateClient.tryConsumeMemoryPrefetch('initial', 100); + setTimeout(() => privateClient.cancelPendingMemoryPrefetch('abort'), 10); + await vi.advanceTimersByTimeAsync(10); + + await expect(consume).resolves.toBeNull(); + expect(controller.signal.aborted).toBe(true); + expect(client['pendingMemoryPrefetch']).toBeUndefined(); + }); + + it('should not consume a prefetch replaced during the bounded wait', async () => { + vi.useFakeTimers(); + type RecallResult = { + prompt: string; + selectedDocs: Array<{ + type: 'user'; + filePath: string; + relativePath: string; + filename: string; + title: string; + description: string; + body: string; + mtimeMs: number; + }>; + strategy: 'model'; + }; + let settleRecall: ((value: RecallResult) => void) | undefined; + const handle = { + promise: new Promise((resolve) => { + settleRecall = resolve; + }), + settledAt: null as number | null, + result: null, + consumed: false, + terminalLogged: false, + fastResultRef: { current: null }, + fastDelivered: false, + fastDeliveredPaths: new Set(), + firedAt: Date.now(), + controller: new AbortController(), + }; + client['pendingMemoryPrefetch'] = handle; + const privateClient = client as unknown as { + tryConsumeMemoryPrefetch: ( + deliveryPoint: 'initial', + waitMs: number, + ) => Promise; + }; + + const consume = privateClient.tryConsumeMemoryPrefetch('initial', 100); + + // The handle is replaced mid-wait and only settles afterwards; the + // post-wait guard must refuse the stale handle instead of consuming + // it. + const replacement = { ...handle, controller: new AbortController() }; + setTimeout(() => { + client['pendingMemoryPrefetch'] = replacement; + }, 10); + setTimeout(() => { + handle.settledAt = Date.now(); + settleRecall!({ + prompt: '## Relevant memory\n\nReplaced result.', + selectedDocs: [], + strategy: 'model', + }); + }, 20); + + await vi.advanceTimersByTimeAsync(100); + + await expect(consume).resolves.toBeNull(); + expect(handle.consumed).toBe(false); + expect(client['pendingMemoryPrefetch']).toBe(replacement); + }); + + it('should not apply the initial wait budget on Cron turns', async () => { + // Cron recall fires too, but its consume point is zero-wait: with a + // never-settling recall the Cron request must proceed at elapsed 0 + // instead of being held for the user-query budget. + vi.useFakeTimers(); + mockMemoryManager.recall.mockReturnValue(new Promise(() => {})); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Cron response' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'Scheduled sweep' }], + new AbortController().signal, + 'prompt-id-cron-memory', + { type: SendMessageType.Cron }, + ), + ); + + // Zero elapsed: the Cron turn must not be held by the recall budget. + await vi.advanceTimersByTimeAsync(0); + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.not.arrayContaining([ + expect.stringContaining('Relevant memory'), + ]), + expect.any(AbortSignal), + ); + await done; + }); + + it('should keep the ToolResult consume point zero-wait', async () => { + // The ToolResult delivery point must never block on the recall + // budget: with a still-pending prefetch the ToolResult turn proceeds + // at elapsed 0 and without memory. + vi.useFakeTimers(); + client['pendingMemoryPrefetch'] = { + promise: new Promise(() => {}), + settledAt: null, + result: null, + consumed: false, + terminalLogged: false, + fastResultRef: { current: null }, + fastDelivered: false, + fastDeliveredPaths: new Set(), + firedAt: Date.now(), + controller: new AbortController(), + }; + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'tool result turn' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ functionResponse: { name: 'foo', response: { ok: true } } }], + new AbortController().signal, + 'prompt-id-tool-result-zero-wait', + { type: SendMessageType.ToolResult }, + ), + ); + + await vi.advanceTimersByTimeAsync(0); + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.not.arrayContaining([ + expect.stringContaining('Relevant memory'), + ]), + expect.any(AbortSignal), + ); + await done; + }); + it('should abort the previous prefetch when a new UserQuery arrives mid-flight', async () => { // Pending recall on first UserQuery — never resolves on its own. const abortSignals: AbortSignal[] = []; @@ -6757,6 +7667,9 @@ hello result, consumed: false, terminalLogged: false, + fastResultRef: { current: null }, + fastDelivered: false, + fastDeliveredPaths: new Set(), firedAt: Date.now(), controller: new AbortController(), }; diff --git a/packages/core/src/core/client.ts b/packages/core/src/core/client.ts index cf7e0c7d2a2..736d1dd0425 100644 --- a/packages/core/src/core/client.ts +++ b/packages/core/src/core/client.ts @@ -82,6 +82,7 @@ import type { UserPromptRecordPayload } from '../services/chatRecordingService.j // Tools import type { RelevantAutoMemoryPromptResult } from '../memory/manager.js'; import { AUTO_SKILL_THRESHOLD } from '../memory/manager.js'; +import { buildRelevantAutoMemoryPrompt } from '../memory/recall.js'; import { isManagedMemoryPath } from '../memory/paths.js'; import { isProjectSkillPath } from '../skills/skill-paths.js'; import { ToolNames } from '../tools/tool-names.js'; @@ -170,6 +171,7 @@ import { PermissionMode, type StopHookOutput } from '../hooks/types.js'; const MAX_TURNS = 100; const MAX_RECENT_TOOL_NAMES_FOR_MEMORY = 20; +const INITIAL_MEMORY_RECALL_WAIT_MS = 100; export enum SendMessageType { UserQuery = 'userQuery', @@ -296,12 +298,21 @@ function sameActiveGoalProjection( * Lifecycle: * 1. Created on UserQuery/Cron — the recall promise fires immediately, * `pendingMemoryPrefetch` is set to this handle. - * 2. Consumed at either of two opportunistic points: a zero-wait - * `settledAt !== null` poll just before the UserQuery main request, - * or — if recall hadn't settled yet — on the first ToolResult turn. + * 2. Consumed at either of two points: a bounded wait just before the + * UserQuery main request, or — if recall remains pending — on the first + * ToolResult turn. * 3. Aborted-and-discarded by every cleanup path (resetChat, * MaxSessionTurns, etc.) or replaced when a new UserQuery arrives. */ +/** + * Publication slot for recall's deterministic result, plus a one-shot + * listener for its arrival. + */ +type MemoryFastResultBox = { + current: RelevantAutoMemoryPromptResult | null; + onArrive?: () => void; +}; + type MemoryPrefetchHandle = { promise: Promise; /** Set by promise.finally(). null until the promise settles. */ @@ -314,6 +325,19 @@ type MemoryPrefetchHandle = { terminalLogged: boolean; firedAt: number; controller: AbortController; + /** + * Deterministic result published by recall before it blocks on the model + * selector. A box rather than a plain field because recall can invoke the + * callback before this handle object exists. + * + * `onArrive` lets the bounded initial wait stop as soon as there is + * something to deliver, instead of always spending the whole budget. + */ + fastResultRef: MemoryFastResultBox; + /** True after the fast result was injected — prevents double-inject and double-log. */ + fastDelivered: boolean; + /** Paths injected by the fast phase, excluded from the later refined delivery. */ + fastDeliveredPaths: Set; }; /** Tools that can write to the skills directory, used to detect skillsModifiedInSession. */ @@ -847,11 +871,25 @@ export class GeminiClient { handle: MemoryPrefetchHandle, discardReason: MemoryRecallDiscardReason, ): void { + const result = handle.result ?? EMPTY_RELEVANT_AUTO_MEMORY_RESULT; + // A settled result whose every document the fast phase already injected + // was not lost, whatever ended the turn — most often a tool-free turn + // reaching `no_safe_delivery_point`. Reporting those under the + // cancellation reason would inflate the "memory never reached the model" + // bucket with turns that did get it, so apply the same rule the + // ToolResult consume point uses. A partial overlap still reports the + // cancellation reason: the documents outside `fastDeliveredPaths` + // genuinely had no delivery point. + const everyDocAlreadyDelivered = + result.selectedDocs.length > 0 && + result.selectedDocs.every((doc) => + handle.fastDeliveredPaths.has(doc.filePath), + ); this.logMemoryPrefetchDelivery( handle, 'discarded', - handle.result ?? EMPTY_RELEVANT_AUTO_MEMORY_RESULT, - discardReason, + result, + everyDocAlreadyDelivered ? 'already_delivered' : discardReason, ); } @@ -869,37 +907,146 @@ export class GeminiClient { } /** - * Atomically consume the pending prefetch if it has already settled. - * Returns the recall result (caller decides where to inject it in - * `requestToSend`), or `null` if there's nothing to consume yet. + * Atomically consume the pending prefetch, optionally waiting for a bounded + * initial-turn budget. Budget expiry leaves the recall running for the next + * safe delivery point. * * Centralises the consume-and-mark dance so the UserQuery and ToolResult * inject sites can't drift on the guard logic. */ private async tryConsumeMemoryPrefetch( deliveryPoint: Exclude, + waitMs = 0, ): Promise { const handle = this.pendingMemoryPrefetch; - if (!handle || handle.settledAt === null || handle.consumed) { + if (!handle || handle.consumed) { return null; } + + // `waitMs` is a ceiling, not a fixed cost. The wait ends on whichever + // comes first: recall settling, the deterministic result being published, + // cancellation, or the budget expiring. + // + // Ending on the fast result matters more than it looks. That result is + // published once recall has scanned the memory tree, which is milliseconds + // for an ordinary tree — while the model selector is a network round trip + // that this design already assumes will miss the budget. Spending the rest + // of the budget after the fast result is in hand therefore buys an + // outcome that almost never arrives, and charges every user turn for it. + // See `recall-scan-latency.test.ts` for the scan measurements. + // + // Consequence worth stating plainly, because the branch below reads as + // if it still arbitrated: on the initial turn, once the deterministic + // scorer matches anything, the fast result wins — the selector's speed is + // irrelevant. `onFastResult` is published before recall even issues the + // selector request, so `settledAt` is necessarily null when the wait ends + // on it. The settled-recall branch is reached at this point only when no + // fast result exists at all: no `Config`, or nothing matched + // lexically. That is deliberate, not incidental — a model side query does + // not complete inside this ceiling, so arbitrating between them would + // cost every turn the remainder of the budget to win a race that does not + // happen. The selector's judgement reaches the model at the ToolResult + // delivery point instead. Pinned by "delivers the fast result even when + // the selector settles inside the budget". + if ( + handle.settledAt === null && + handle.fastResultRef.current === null && + waitMs > 0 + ) { + await new Promise((resolve) => { + const finish = () => { + clearTimeout(timer); + handle.controller.signal.removeEventListener('abort', finish); + if (handle.fastResultRef.onArrive === finish) { + handle.fastResultRef.onArrive = undefined; + } + resolve(); + }; + + const timer = setTimeout(finish, waitMs); + if (handle.controller.signal.aborted) { + finish(); + } else { + handle.controller.signal.addEventListener('abort', finish, { + once: true, + }); + handle.fastResultRef.onArrive = finish; + void handle.promise.then(finish, finish); + } + }); + } + + if (this.pendingMemoryPrefetch !== handle || handle.consumed) { + return null; + } + + // Budget expired with the selector still in flight. Inject the + // deterministic result now rather than gambling on a later tool call: + // a turn that makes none has no safe delivery point at all. The handle + // stays pending so the model-selected result can still land later. + if (handle.settledAt === null) { + if (deliveryPoint !== 'initial' || handle.fastDelivered) { + return null; + } + const fast = handle.fastResultRef.current; + if (!fast?.prompt) { + return null; + } + handle.fastDelivered = true; + for (const doc of fast.selectedDocs) { + this.surfacedRelevantAutoMemoryPaths.add(doc.filePath); + handle.fastDeliveredPaths.add(doc.filePath); + } + logMemoryRecallDelivery( + this.config, + new MemoryRecallDeliveryEvent({ + phase: 'fast', + delivery_point: 'initial', + strategy: fast.strategy, + docs_selected: fast.selectedDocs.length, + latency_ms: Date.now() - handle.firedAt, + }), + ); + return fast; + } + handle.consumed = true; this.pendingMemoryPrefetch = undefined; const result = await handle.promise; // already settled, returns immediately - if (result.prompt) { - for (const doc of result.selectedDocs) { + // Drop anything the fast phase already put in front of the model. Both + // results come from the same scan, so the selector never saw the fast + // documents as excluded and can legitimately re-select them. + const remainingDocs = result.selectedDocs.filter( + (doc) => !handle.fastDeliveredPaths.has(doc.filePath), + ); + const deduped = + remainingDocs.length === result.selectedDocs.length + ? result + : { + ...result, + selectedDocs: remainingDocs, + prompt: + remainingDocs.length > 0 + ? buildRelevantAutoMemoryPrompt(remainingDocs) + : '', + }; + + if (deduped.prompt) { + for (const doc of deduped.selectedDocs) { this.surfacedRelevantAutoMemoryPaths.add(doc.filePath); } - this.logMemoryPrefetchDelivery(handle, deliveryPoint, result); + this.logMemoryPrefetchDelivery(handle, deliveryPoint, deduped); } else { this.logMemoryPrefetchDelivery( handle, 'discarded', result, - 'no_relevant_results', + result.selectedDocs.length > 0 + ? 'already_delivered' + : 'no_relevant_results', ); } - return result; + return deduped; } async resetChat(): Promise { @@ -2760,6 +2907,7 @@ export class GeminiClient { } else { signal.addEventListener('abort', onParentAbort, { once: true }); } + const fastResultRef: MemoryFastResultBox = { current: null }; const promise = this.config .getMemoryManager() .recall( @@ -2770,6 +2918,10 @@ export class GeminiClient { excludedFilePaths: this.surfacedRelevantAutoMemoryPaths, recentTools: [...this.recentCompletedToolNames], abortSignal: controller.signal, + onFastResult: (result) => { + fastResultRef.current = result; + fastResultRef.onArrive?.(); + }, }, ) .catch((error: unknown) => { @@ -2800,6 +2952,9 @@ export class GeminiClient { terminalLogged: false, firedAt: Date.now(), controller, + fastResultRef, + fastDelivered: false, + fastDeliveredPaths: new Set(), }; void promise.then((result) => { handle.result = result; @@ -3127,14 +3282,12 @@ export class GeminiClient { } } - // Zero-wait poll: consume only if the prefetch has already settled. - // Done AFTER the async reminder setup above so recall settling during - // those awaits still gets caught here. (settledAt is set in - // promise.finally(); microtask ordering guarantees it's visible - // after any await prior to this point — flatMapTextParts above is - // the natural drain.) If still not settled, skip — the ToolResult - // inject point will retry on the next turn. - const userQueryMemory = await this.tryConsumeMemoryPrefetch('initial'); + const userQueryMemory = await this.tryConsumeMemoryPrefetch( + 'initial', + messageType === SendMessageType.UserQuery + ? INITIAL_MEMORY_RECALL_WAIT_MS + : 0, + ); if (userQueryMemory?.prompt) { // Unshift to the front of systemReminders: on a UserQuery turn // requestToSend leads with user text, so positioning memory at diff --git a/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json new file mode 100644 index 00000000000..8adf013b547 --- /dev/null +++ b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json @@ -0,0 +1,526 @@ +{ + "docs": [ + { + "id": "en-release", + "type": "project", + "title": "Release process", + "description": "Production deployment checklist", + "body": "Verify monitoring and rollback switches before shipping." + }, + { + "id": "en-style", + "type": "user", + "title": "Response preferences", + "description": "Concise answer style", + "body": "Keep explanations direct and skip trailing summaries." + }, + { + "id": "en-oncall", + "type": "reference", + "title": "Oncall latency board", + "description": "Grafana service dashboard", + "body": "Service response times and alert entry points." + }, + { + "id": "en-rollback", + "type": "feedback", + "title": "Operational notes", + "description": "Miscellaneous guidance", + "body": "Emergency rollback procedures require owner approval." + }, + { + "id": "en-testing", + "type": "feedback", + "title": "Testing rules", + "description": "Run focused vitest suites", + "body": "Avoid full builds on constrained machines." + }, + { + "id": "en-migration", + "type": "project", + "title": "Database migration", + "description": "Postgres schema rollout", + "body": "Run migrations behind a feature flag." + }, + { + "id": "en-owner", + "type": "user", + "title": "Team ownership", + "description": "Platform team contacts", + "body": "Escalate storage issues to the platform team." + }, + { + "id": "en-secrets", + "type": "reference", + "title": "Credential locations", + "description": "Vault paths for staging", + "body": "Staging credentials live in the vault under platform." + }, + { + "id": "en-empty", + "type": "project", + "title": "Sprint placeholder", + "description": "Upcoming milestone", + "body": "_No entries yet._" + }, + { + "id": "zh-deploy", + "type": "project", + "title": "生产部署流程", + "description": "发布检查清单", + "body": "上线前确认监控和回滚开关。" + }, + { + "id": "zh-api", + "type": "reference", + "title": "接口延迟排查", + "description": "API 性能看板", + "body": "记录服务响应时间和告警入口。" + }, + { + "id": "zh-auth", + "type": "project", + "title": "认证配置", + "description": "登录问题排查", + "body": "会话过期和权限配置检查。" + }, + { + "id": "zh-perf", + "type": "project", + "title": "性能优化目标", + "description": "季度性能指标", + "body": "降低冷启动时间。" + }, + { + "id": "ja-auth", + "type": "project", + "title": "認証設定ガイド", + "description": "ユーザーログイン構成", + "body": "セッション設定の確認手順。" + }, + { + "id": "ja-deploy", + "type": "reference", + "title": "デプロイ手順", + "description": "リリース運用", + "body": "本番反映前の確認事項。" + }, + { + "id": "ja-hiragana", + "type": "user", + "title": "よくあるしつもん", + "description": "ひらがなだけでかいたあんない", + "body": "ひらがなのとうこうにそなえたきろく。" + }, + { + "id": "ja-review", + "type": "feedback", + "title": "レビュー方針", + "description": "コードレビューの基準", + "body": "小さな変更に分割する。" + }, + { + "id": "ko-deploy", + "type": "project", + "title": "배포 절차", + "description": "릴리스 체크리스트", + "body": "운영 반영 전에 모니터링을 확인한다." + }, + { + "id": "ko-auth", + "type": "reference", + "title": "인증 설정", + "description": "로그인 문제 해결", + "body": "세션 만료와 권한 구성을 확인한다." + }, + { + "id": "ko-perf", + "type": "project", + "title": "성능 목표", + "description": "분기 성능 지표", + "body": "콜드 스타트 시간을 줄인다." + }, + { + "id": "mixed-api", + "type": "reference", + "title": "Qwen API 限流", + "description": "Rate limit dashboard", + "body": "检查 quota 和请求速率。" + }, + { + "id": "mixed-deploy", + "type": "project", + "title": "Deployment 发布说明", + "description": "Release 流程", + "body": "Mixed-language deployment notes." + }, + { + "id": "ru-deploy", + "type": "project", + "title": "Процесс развёртывания", + "description": "Контрольный список релиза", + "body": "Перед выкатом проверить мониторинг и переключатели отката." + }, + { + "id": "el-auth", + "type": "reference", + "title": "Ρύθμιση ταυτοποίησης", + "description": "Επίλυση προβλημάτων σύνδεσης", + "body": "Έλεγχος λήξης συνεδρίας και δικαιωμάτων." + }, + { + "id": "fr-perf", + "type": "project", + "title": "Réduction du démarrage à froid", + "description": "Objectifs de performance trimestriels", + "body": "Mesurer la latence côté serveur avant toute optimisation." + } + ], + "cases": [ + { + "id": "en-release-process", + "category": "english", + "query": "release process", + "relevantIds": ["en-release"], + "expectedTopId": "en-release" + }, + { + "id": "en-production-deployment", + "category": "english", + "query": "production deployment checklist", + "relevantIds": ["en-release"], + "expectedTopId": "en-release" + }, + { + "id": "en-concise-style", + "category": "english", + "query": "concise answer style", + "relevantIds": ["en-style"], + "expectedTopId": "en-style" + }, + { + "id": "en-response-preferences", + "category": "english", + "query": "response preferences", + "relevantIds": ["en-style"], + "expectedTopId": "en-style" + }, + { + "id": "en-latency-board", + "category": "english", + "query": "oncall latency board", + "relevantIds": ["en-oncall"], + "expectedTopId": "en-oncall" + }, + { + "id": "en-grafana-dashboard", + "category": "english", + "query": "grafana service dashboard", + "relevantIds": ["en-oncall"], + "expectedTopId": "en-oncall" + }, + { + "id": "en-vitest-suites", + "category": "english", + "query": "focused vitest suites", + "relevantIds": ["en-testing"], + "expectedTopId": "en-testing" + }, + { + "id": "en-database-migration", + "category": "english", + "query": "database migration", + "relevantIds": ["en-migration"], + "expectedTopId": "en-migration" + }, + { + "id": "en-postgres-schema", + "category": "english", + "query": "postgres schema rollout", + "relevantIds": ["en-migration"], + "expectedTopId": "en-migration" + }, + { + "id": "en-credential-locations", + "category": "english", + "query": "credential locations", + "relevantIds": ["en-secrets"], + "expectedTopId": "en-secrets" + }, + { + "id": "en-team-ownership", + "category": "english", + "query": "team ownership", + "relevantIds": ["en-owner"], + "expectedTopId": "en-owner" + }, + { + "id": "zh-deploy-title", + "category": "chinese", + "query": "生产部署", + "relevantIds": ["zh-deploy"], + "expectedTopId": "zh-deploy" + }, + { + "id": "zh-deploy-description", + "category": "chinese", + "query": "发布检查", + "relevantIds": ["zh-deploy"], + "expectedTopId": "zh-deploy" + }, + { + "id": "zh-api-title", + "category": "chinese", + "query": "接口延迟", + "relevantIds": ["zh-api"], + "expectedTopId": "zh-api" + }, + { + "id": "zh-api-troubleshooting", + "category": "chinese", + "query": "延迟排查", + "relevantIds": ["zh-api"], + "expectedTopId": "zh-api" + }, + { + "id": "zh-auth-config", + "category": "chinese", + "query": "认证配置", + "relevantIds": ["zh-auth"], + "expectedTopId": "zh-auth" + }, + { + "id": "zh-performance-goal", + "category": "chinese", + "query": "性能优化目标", + "relevantIds": ["zh-perf"], + "expectedTopId": "zh-perf" + }, + { + "id": "ja-auth-title", + "category": "japanese", + "query": "認証設定", + "relevantIds": ["ja-auth"], + "expectedTopId": "ja-auth" + }, + { + "id": "ja-login-description", + "category": "japanese", + "query": "ログイン構成", + "relevantIds": ["ja-auth"], + "expectedTopId": "ja-auth" + }, + { + "id": "ja-deploy-title", + "category": "japanese", + "query": "デプロイ手順", + "relevantIds": ["ja-deploy"], + "expectedTopId": "ja-deploy" + }, + { + "id": "ja-release-description", + "category": "japanese", + "query": "リリース運用", + "relevantIds": ["ja-deploy"], + "expectedTopId": "ja-deploy" + }, + { + "id": "ja-hiragana-only", + "category": "japanese", + "query": "よくあるしつもん", + "relevantIds": ["ja-hiragana"], + "expectedTopId": "ja-hiragana" + }, + { + "id": "ja-review-standard", + "category": "japanese", + "query": "コードレビューの基準", + "relevantIds": ["ja-review"], + "expectedTopId": "ja-review" + }, + { + "id": "ko-deploy-title", + "category": "korean", + "query": "배포 절차", + "relevantIds": ["ko-deploy"], + "expectedTopId": "ko-deploy" + }, + { + "id": "ko-release-checklist", + "category": "korean", + "query": "릴리스 체크", + "relevantIds": ["ko-deploy"], + "expectedTopId": "ko-deploy" + }, + { + "id": "ko-auth-title", + "category": "korean", + "query": "인증 설정", + "relevantIds": ["ko-auth"], + "expectedTopId": "ko-auth" + }, + { + "id": "ko-login-description", + "category": "korean", + "query": "로그인 문제", + "relevantIds": ["ko-auth"], + "expectedTopId": "ko-auth" + }, + { + "id": "ko-performance-goal", + "category": "korean", + "query": "성능 목표", + "relevantIds": ["ko-perf"], + "expectedTopId": "ko-perf" + }, + { + "id": "mixed-rate-limit-han", + "category": "mixed", + "query": "qwen api 限流", + "relevantIds": ["mixed-api"], + "expectedTopId": "mixed-api" + }, + { + "id": "mixed-rate-limit-ascii", + "category": "mixed", + "query": "rate limit dashboard", + "relevantIds": ["mixed-api"], + "expectedTopId": "mixed-api" + }, + { + "id": "mixed-deployment-han", + "category": "mixed", + "query": "deployment 发布说明", + "relevantIds": ["mixed-deploy"], + "expectedTopId": "mixed-deploy" + }, + { + "id": "nfkc-fullwidth-qwen-api", + "category": "nfkc", + "query": "ＱＷＥＮ ＡＰＩ", + "relevantIds": ["mixed-api"], + "expectedTopId": "mixed-api" + }, + { + "id": "nfkc-fullwidth-production", + "category": "nfkc", + "query": "ＰＲＯＤＵＣＴＩＯＮ deployment", + "relevantIds": ["en-release"], + "expectedTopId": "en-release" + }, + { + "id": "body-only-owner-approval", + "category": "body-only", + "query": "owner approval", + "relevantIds": ["en-rollback"], + "expectedTopId": "en-rollback" + }, + { + "id": "body-only-feature-flag", + "category": "body-only", + "query": "feature flag", + "relevantIds": ["en-migration"], + "expectedTopId": "en-migration" + }, + { + "id": "body-only-alert-entry", + "category": "body-only", + "query": "alert entry points", + "relevantIds": ["en-oncall"], + "expectedTopId": "en-oncall" + }, + { + "id": "no-result-kubernetes-ingress", + "category": "no-result", + "query": "kubernetes ingress certificate", + "relevantIds": [] + }, + { + "id": "no-result-quantum", + "category": "no-result", + "query": "quantum entanglement", + "relevantIds": [] + }, + { + "id": "no-result-knowledge-graph", + "category": "no-result", + "query": "knowledge graph embeddings", + "relevantIds": [] + }, + { + "id": "no-result-weather", + "category": "no-result", + "query": "weather forecast tomorrow", + "relevantIds": [] + }, + { + "id": "no-result-marketing", + "category": "no-result", + "query": "marketing budget spreadsheet", + "relevantIds": [] + }, + { + "id": "no-result-single-han", + "category": "no-result", + "query": "部", + "relevantIds": [] + }, + { + "id": "no-result-single-kanji", + "category": "no-result", + "query": "認", + "relevantIds": [] + }, + { + "id": "no-result-single-hangul", + "category": "no-result", + "query": "배", + "relevantIds": [] + }, + { + "id": "no-result-short-ascii", + "category": "no-result", + "query": "go", + "relevantIds": [] + }, + { + "id": "semantic-terse-replies", + "category": "semantic-no-lexical", + "query": "stop padding your replies with recaps", + "relevantIds": ["en-style"] + }, + { + "id": "semantic-latency-chart", + "category": "semantic-no-lexical", + "query": "which chart shows p99 spikes", + "relevantIds": ["en-oncall"] + }, + { + "id": "semantic-slow-first-paint", + "category": "semantic-no-lexical", + "query": "首屏加载太久了", + "relevantIds": ["zh-perf"] + }, + { + "id": "other-script-ru-deploy", + "category": "other-script", + "query": "процесс развёртывания", + "relevantIds": ["ru-deploy"], + "expectedTopId": "ru-deploy" + }, + { + "id": "other-script-el-auth", + "category": "other-script", + "query": "ρύθμιση ταυτοποίησης", + "relevantIds": ["el-auth"], + "expectedTopId": "el-auth" + }, + { + "id": "other-script-fr-perf", + "category": "other-script", + "query": "démarrage à froid", + "relevantIds": ["fr-perf"], + "expectedTopId": "fr-perf" + } + ] +} diff --git a/packages/core/src/memory/memoryLifecycle.integration.test.ts b/packages/core/src/memory/memoryLifecycle.integration.test.ts index 6318306ab68..971656e70fb 100644 --- a/packages/core/src/memory/memoryLifecycle.integration.test.ts +++ b/packages/core/src/memory/memoryLifecycle.integration.test.ts @@ -229,4 +229,65 @@ describe('managed auto-memory lifecycle integration', () => { expect(recall.prompt).toContain('user/'); expect(recall.prompt).toContain('reference/'); }); + + it('recalls a relevant topic beyond the general 200-document scan cap', async () => { + const referenceDir = path.dirname( + getAutoMemoryFilePath(projectRoot, 'reference/filler-000.md'), + ); + await fs.mkdir(referenceDir, { recursive: true }); + await Promise.all( + Array.from({ length: 200 }, (_, index) => + fs.writeFile( + path.join( + referenceDir, + `filler-${String(index).padStart(3, '0')}.md`, + ), + [ + '---', + 'type: reference', + `name: Filler ${index}`, + 'description: Unrelated historical note', + '---', + '', + 'No matching content.', + ].join('\n'), + 'utf-8', + ), + ), + ); + + const targetPath = getAutoMemoryFilePath( + projectRoot, + 'reference/overflow-target.md', + ); + await fs.writeFile( + targetPath, + [ + '---', + 'type: reference', + 'name: Overflow Zephyr Marker', + 'description: Unique recall target beyond the general scan cap', + '---', + '', + 'The saved codeword is OVERFLOW-ZEPHYR-7040.', + ].join('\n'), + 'utf-8', + ); + await fs.utimes(targetPath, new Date(0), new Date(0)); + + const cappedDocs = await scanAutoMemoryTopicDocuments(projectRoot); + expect(cappedDocs).toHaveLength(200); + expect(cappedDocs.some((doc) => doc.filePath === targetPath)).toBe(false); + + const recall = await resolveRelevantAutoMemoryPromptForQuery( + projectRoot, + 'What is the overflow zephyr codeword?', + ); + + expect(recall.strategy).toBe('heuristic'); + expect(recall.selectedDocs.map((doc) => doc.filePath)).toContain( + targetPath, + ); + expect(recall.prompt).toContain('OVERFLOW-ZEPHYR-7040'); + }); }); diff --git a/packages/core/src/memory/recall-delivery-eval.test.ts b/packages/core/src/memory/recall-delivery-eval.test.ts new file mode 100644 index 00000000000..8cc39ec73dc --- /dev/null +++ b/packages/core/src/memory/recall-delivery-eval.test.ts @@ -0,0 +1,496 @@ +/** + * @license + * Copyright 2025 Google LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { + MAX_FAST_RECALL_DOCS, + selectRelevantAutoMemoryDocuments, +} from './recall.js'; +import type { ScannedAutoMemoryDocument } from './scan.js'; +import type { AutoMemoryType } from './types.js'; + +/** + * Delivery-stage measurement for the deterministic fast path. + * + * `recall-eval.test.ts` answers "did recall pick the right documents". This + * file answers the separate question the fast path exists for: "did the + * picked documents actually reach the model at a safe delivery point". A + * perfect selector that never gets delivered is worth nothing. + * + * ## What is measured and what is modelled + * + * Measured for real: the deterministic scorer's own latency, which documents + * each design delivers, and the overlap between the fast and refined sets + * under the shipped dedupe rule. + * + * Modelled, not measured: the model selector's latency. It is a network side + * query and cannot be timed in a unit test, so it is a parameter here. The + * conclusions are reported per latency scenario rather than as a single + * number, and the only structural claim — that a selector slower than the + * initial budget leaves a tool-free turn with no delivery point at all under + * the single-path design — holds for every scenario above the budget. + */ + +const fixtureUrl = new URL( + './__fixtures__/auto-memory-recall-eval.json', + import.meta.url, +); + +/** Mirrors INITIAL_MEMORY_RECALL_WAIT_MS in client.ts. */ +const INITIAL_BUDGET_MS = 100; +const RECALL_AT = 5; + +interface EvalDoc { + id: string; + type: AutoMemoryType; + title: string; + description: string; + body: string; +} + +interface EvalCase { + id: string; + category: string; + query: string; + relevantIds: string[]; + expectedTopId?: string; +} + +interface EvalFixture { + docs: EvalDoc[]; + cases: EvalCase[]; +} + +/** + * Selector latency scenarios in milliseconds. 40 ms stands for a warm, + * unusually fast round trip that lands inside the budget; the rest span an + * ordinary to slow model call. Only the first is inside INITIAL_BUDGET_MS. + */ +const SELECTOR_LATENCY_SCENARIOS_MS = [40, 250, 600, 1500, 3000] as const; + +/** Whether the first turn issues a tool call, which is the only later delivery point. */ +type TurnShape = 'tool-free' | 'tool-using'; + +interface DeliveryOutcome { + /** Documents in front of the model in the very first request. */ + initialDocIds: string[]; + /** Documents added at the first ToolResult, if any. */ + toolResultDocIds: string[]; + /** Documents selected but never delivered anywhere. */ + discardedDocIds: string[]; + /** Documents delivered more than once across both points. */ + duplicateDocIds: string[]; +} + +function loadFixture(): EvalFixture { + return JSON.parse(readFileSync(fixtureUrl, 'utf8')) as EvalFixture; +} + +function toScannedDocs(docs: EvalDoc[]): ScannedAutoMemoryDocument[] { + return docs.map((doc) => ({ + type: doc.type, + filePath: `/memory/${doc.id}.md`, + relativePath: `${doc.id}.md`, + filename: `${doc.id}.md`, + title: doc.title, + description: doc.description, + body: doc.body, + mtimeMs: 1, + })); +} + +const docIdOf = (doc: ScannedAutoMemoryDocument) => + doc.filename.replace(/\.md$/, ''); + +/** + * Stand-in for the model selector's choice. The model is unavailable here, so + * the deterministic top-5 is used. For the duplicate-rate question this is the + * worst case on purpose: the fast set is drawn from the same ranking, so + * overlap is maximal and the dedupe rule gets the hardest input it can get. + */ +function refinedSelection( + query: string, + docs: ScannedAutoMemoryDocument[], +): ScannedAutoMemoryDocument[] { + return selectRelevantAutoMemoryDocuments(query, docs, RECALL_AT); +} + +function fastSelection( + query: string, + docs: ScannedAutoMemoryDocument[], +): ScannedAutoMemoryDocument[] { + return selectRelevantAutoMemoryDocuments(query, docs, RECALL_AT).slice( + 0, + MAX_FAST_RECALL_DOCS, + ); +} + +/** + * Single-path design as shipped in #8716 before the fast path: one result, + * delivered initially only if the selector settles inside the budget, + * otherwise held for a ToolResult that a tool-free turn never produces. + */ +function simulateSinglePath( + query: string, + docs: ScannedAutoMemoryDocument[], + selectorLatencyMs: number, + turnShape: TurnShape, +): DeliveryOutcome { + const refined = refinedSelection(query, docs).map(docIdOf); + if (refined.length === 0) { + return { + initialDocIds: [], + toolResultDocIds: [], + discardedDocIds: [], + duplicateDocIds: [], + }; + } + if (selectorLatencyMs <= INITIAL_BUDGET_MS) { + return { + initialDocIds: refined, + toolResultDocIds: [], + discardedDocIds: [], + duplicateDocIds: [], + }; + } + if (turnShape === 'tool-using') { + // The simulation models the safe ToolResult point after selector completion. + // Selector latency is varied by scenario; ToolResult timing is not. + return { + initialDocIds: [], + toolResultDocIds: refined, + discardedDocIds: [], + duplicateDocIds: [], + }; + } + return { + initialDocIds: [], + toolResultDocIds: [], + discardedDocIds: refined, + duplicateDocIds: [], + }; +} + +/** + * Fast-path design: same refined result, plus a deterministic result injected + * at budget expiry and excluded from the later refined delivery. + */ +function simulateFastPath( + query: string, + docs: ScannedAutoMemoryDocument[], + selectorLatencyMs: number, + turnShape: TurnShape, +): DeliveryOutcome { + const refined = refinedSelection(query, docs).map(docIdOf); + if (selectorLatencyMs <= INITIAL_BUDGET_MS) { + // Selector won the race; the fast result is never consumed. + return { + initialDocIds: refined, + toolResultDocIds: [], + discardedDocIds: [], + duplicateDocIds: [], + }; + } + + const fast = fastSelection(query, docs).map(docIdOf); + const delivered = new Set(fast); + const remaining = refined.filter((id) => !delivered.has(id)); + + if (turnShape === 'tool-using') { + // The simulation models the safe ToolResult point after selector completion. + // Selector latency is varied by scenario; ToolResult timing is not. + return { + initialDocIds: fast, + toolResultDocIds: remaining, + discardedDocIds: [], + duplicateDocIds: fast.filter((id) => remaining.includes(id)), + }; + } + return { + initialDocIds: fast, + toolResultDocIds: [], + discardedDocIds: remaining, + duplicateDocIds: [], + }; +} + +interface DeliverySummary { + answerableCases: number; + /** Share of answerable cases with at least one document in the first request. */ + firstTurnDeliveryRate: number; + /** Same, restricted to turns that make no tool call. */ + toolFreeFirstTurnDeliveryRate: number; + /** Share of answerable cases where the relevant document reached the model at all. */ + anyDeliveryRate: number; + /** + * Same, on a turn that does make a tool call. The single-path design is not + * broken here — it delivers one request later — so reporting this keeps the + * comparison honest about where the actual gap is. + */ + anyDeliveryRateToolUsing: number; + /** Share of answerable cases delivering the same document twice. */ + duplicateDeliveryRate: number; + /** + * Share of answerable cases where the fast and refined sets overlap at all. + * This is what the dedupe rule has to suppress; without it these cases would + * become duplicate deliveries. + */ + overlapBeforeDedupeRate: number; +} + +function summarize( + fixture: EvalFixture, + simulate: typeof simulateSinglePath, + selectorLatencyMs: number, + filter: (testCase: EvalCase) => boolean = () => true, +): DeliverySummary { + const docs = toScannedDocs(fixture.docs); + const answerable = fixture.cases.filter( + (testCase) => testCase.relevantIds.length > 0 && filter(testCase), + ); + + let firstTurnHits = 0; + let toolFreeHits = 0; + let anyDeliveryHits = 0; + let anyDeliveryToolUsingHits = 0; + let duplicateCases = 0; + let overlapCases = 0; + + for (const testCase of answerable) { + const toolFree = simulate( + testCase.query, + docs, + selectorLatencyMs, + 'tool-free', + ); + const toolUsing = simulate( + testCase.query, + docs, + selectorLatencyMs, + 'tool-using', + ); + + if (toolUsing.initialDocIds.length > 0) firstTurnHits += 1; + if (toolFree.initialDocIds.length > 0) toolFreeHits += 1; + if (toolFree.initialDocIds.length + toolFree.toolResultDocIds.length > 0) { + anyDeliveryHits += 1; + } + if ( + toolUsing.initialDocIds.length + toolUsing.toolResultDocIds.length > + 0 + ) { + anyDeliveryToolUsingHits += 1; + } + if (toolUsing.duplicateDocIds.length > 0) duplicateCases += 1; + + // What dedupe had to suppress, independent of which design ran. + const fast = new Set(fastSelection(testCase.query, docs).map(docIdOf)); + if ( + selectorLatencyMs > INITIAL_BUDGET_MS && + refinedSelection(testCase.query, docs).some((doc) => + fast.has(docIdOf(doc)), + ) + ) { + overlapCases += 1; + } + } + + const n = answerable.length; + return { + answerableCases: n, + firstTurnDeliveryRate: firstTurnHits / n, + toolFreeFirstTurnDeliveryRate: toolFreeHits / n, + anyDeliveryRate: anyDeliveryHits / n, + anyDeliveryRateToolUsing: anyDeliveryToolUsingHits / n, + duplicateDeliveryRate: duplicateCases / n, + overlapBeforeDedupeRate: overlapCases / n, + }; +} + +function percentile(sorted: number[], p: number): number { + if (sorted.length === 0) return 0; + const index = Math.min( + sorted.length - 1, + Math.max(0, Math.ceil((p / 100) * sorted.length) - 1), + ); + return sorted[index]; +} + +function measureDeterministicLatencyMs(fixture: EvalFixture): { + p50: number; + p95: number; +} { + const docs = toScannedDocs(fixture.docs); + const samples: number[] = []; + // Warm up so the first-call compile cost doesn't land in the samples. + for (let i = 0; i < 50; i += 1) { + for (const testCase of fixture.cases) { + selectRelevantAutoMemoryDocuments(testCase.query, docs, RECALL_AT); + } + } + for (let i = 0; i < 200; i += 1) { + for (const testCase of fixture.cases) { + const started = performance.now(); + selectRelevantAutoMemoryDocuments(testCase.query, docs, RECALL_AT); + samples.push(performance.now() - started); + } + } + samples.sort((a, b) => a - b); + return { p50: percentile(samples, 50), p95: percentile(samples, 95) }; +} + +/** + * Answerable cases the deterministic scorer can actually reach. The + * `semantic-no-lexical` slice is labeled answerable but shares no token with + * its document, so no fast result exists for it; it is measured on its own + * rather than folded into the delivery guarantee. + */ +const isLexicallyAnswerable = (testCase: EvalCase) => + testCase.category !== 'semantic-no-lexical'; + +const formatPercent = (value: number) => `${(value * 100).toFixed(1)}%`; + +describe('auto-memory recall delivery evaluation', () => { + it('keeps deterministic candidate selection far inside the initial budget', () => { + const { p50, p95 } = measureDeterministicLatencyMs(loadFixture()); + // The fast delivery path reuses this candidate selection, so the measured + // work must stay negligible next to the budget. Generous bound: shared CI. + expect(p95).toBeLessThan(INITIAL_BUDGET_MS / 10); + expect(p50).toBeLessThanOrEqual(p95); + }); + + it('delivers nothing on a tool-free turn under the single-path design when the selector misses the budget', () => { + const fixture = loadFixture(); + for (const latency of SELECTOR_LATENCY_SCENARIOS_MS) { + if (latency <= INITIAL_BUDGET_MS) continue; + const summary = summarize(fixture, simulateSinglePath, latency); + expect(summary.toolFreeFirstTurnDeliveryRate).toBe(0); + expect(summary.anyDeliveryRate).toBe(0); + } + }); + + it('delivers on every tool-free turn whose query the deterministic path can match', () => { + const fixture = loadFixture(); + for (const latency of SELECTOR_LATENCY_SCENARIOS_MS) { + const summary = summarize( + fixture, + simulateFastPath, + latency, + isLexicallyAnswerable, + ); + expect(summary.answerableCases).toBeGreaterThan(0); + expect(summary.toolFreeFirstTurnDeliveryRate).toBe(1); + expect(summary.anyDeliveryRate).toBe(1); + } + }); + + /** + * The bound on the claim above. The fast result is the deterministic + * result, so a query with no lexical match produces no fast result, and a + * tool-free turn asking it still ends with nothing delivered. The fast path + * closes the *timing* gap, not the *matching* gap — only the model selector + * closes the latter, and on a tool-free turn it never lands. + * + * The table below reports the honest overall rate, which is this slice's + * share below 100%, rather than the lexically-answerable rate alone. + */ + it('delivers nothing on a tool-free turn for semantic-only queries', () => { + const fixture = loadFixture(); + for (const latency of SELECTOR_LATENCY_SCENARIOS_MS) { + if (latency <= INITIAL_BUDGET_MS) continue; + const summary = summarize( + fixture, + simulateFastPath, + latency, + (testCase) => !isLexicallyAnswerable(testCase), + ); + expect(summary.answerableCases).toBeGreaterThan(0); + expect(summary.toolFreeFirstTurnDeliveryRate).toBe(0); + expect(summary.anyDeliveryRate).toBe(0); + } + }); + + it('never delivers the same document twice', () => { + const fixture = loadFixture(); + for (const latency of SELECTOR_LATENCY_SCENARIOS_MS) { + expect( + summarize(fixture, simulateFastPath, latency).duplicateDeliveryRate, + ).toBe(0); + } + }); + + it('leaves no-result queries silent under both designs', () => { + const fixture = loadFixture(); + const docs = toScannedDocs(fixture.docs); + for (const testCase of fixture.cases) { + if (testCase.relevantIds.length > 0) continue; + for (const simulate of [simulateSinglePath, simulateFastPath]) { + const outcome = simulate(testCase.query, docs, 600, 'tool-free'); + expect(outcome.initialDocIds).toEqual([]); + expect(outcome.toolResultDocIds).toEqual([]); + } + } + }); + + it('reports the before/after delivery table', () => { + const fixture = loadFixture(); + const { p50, p95 } = measureDeterministicLatencyMs(fixture); + const lines = [ + '', + 'Delivery gate — single path (before) vs deterministic fast path (after)', + `deterministic scoring latency: p50 ${p50.toFixed(3)} ms, p95 ${p95.toFixed(3)} ms (budget ${INITIAL_BUDGET_MS} ms)`, + '', + '| selector latency | metric | before | after |', + '| --- | --- | --- | --- |', + ]; + + for (const latency of SELECTOR_LATENCY_SCENARIOS_MS) { + const before = summarize(fixture, simulateSinglePath, latency); + const after = summarize(fixture, simulateFastPath, latency); + const rows: Array<[string, number, number]> = [ + [ + 'first-turn delivery (tool-using)', + before.firstTurnDeliveryRate, + after.firstTurnDeliveryRate, + ], + [ + 'first-turn delivery (tool-free)', + before.toolFreeFirstTurnDeliveryRate, + after.toolFreeFirstTurnDeliveryRate, + ], + [ + 'delivered at all (tool-free)', + before.anyDeliveryRate, + after.anyDeliveryRate, + ], + [ + 'delivered at all (tool-using)', + before.anyDeliveryRateToolUsing, + after.anyDeliveryRateToolUsing, + ], + [ + 'fast/refined overlap needing dedupe', + before.overlapBeforeDedupeRate, + after.overlapBeforeDedupeRate, + ], + [ + 'duplicate delivery', + before.duplicateDeliveryRate, + after.duplicateDeliveryRate, + ], + ]; + for (const [metric, beforeValue, afterValue] of rows) { + lines.push( + `| ${latency} ms | ${metric} | ${formatPercent(beforeValue)} | ${formatPercent(afterValue)} |`, + ); + } + } + + console.log(lines.join('\n')); + expect(lines.length).toBeGreaterThan(6); + }); +}); diff --git a/packages/core/src/memory/recall-eval.test.ts b/packages/core/src/memory/recall-eval.test.ts new file mode 100644 index 00000000000..d342613c2fa --- /dev/null +++ b/packages/core/src/memory/recall-eval.test.ts @@ -0,0 +1,538 @@ +/** + * @license + * Copyright 2025 Google LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { selectRelevantAutoMemoryDocuments } from './recall.js'; +import type { ScannedAutoMemoryDocument } from './scan.js'; +import type { AutoMemoryType } from './types.js'; + +/** + * Measurement harness for the deterministic recall path. + * + * RFC #7040 gates the multilingual precision change on evidence that English + * Recall@5 and no-result precision do not regress. Behavioural assertions in + * `recall.test.ts` pin individual contracts but cannot answer that question, + * because "no regression" is a statement about the *previous* scorer. This + * file therefore keeps a frozen copy of the pre-change scorer as a reference + * implementation and scores both over one labeled corpus. + * + * Only the deterministic selector is measured. The model selector is the + * normal precision gate and is exercised by the mocked cases in + * `recall.test.ts`; active-tool noise filtering and model-candidate bounding + * likewise stay in that file, since neither is reachable from the pure + * scoring function evaluated here. + */ + +const fixtureUrl = new URL( + './__fixtures__/auto-memory-recall-eval.json', + import.meta.url, +); + +const RECALL_AT = 5; + +const categories = new Set([ + 'english', + 'chinese', + 'japanese', + 'korean', + 'mixed', + 'nfkc', + 'body-only', + 'semantic-no-lexical', + 'other-script', + 'no-result', +] as const); + +type EvalCategory = typeof categories extends Set ? T : never; + +interface EvalDoc { + id: string; + type: AutoMemoryType; + title: string; + description: string; + body: string; +} + +interface EvalCase { + id: string; + category: EvalCategory; + query: string; + relevantIds: string[]; + expectedTopId?: string; +} + +interface EvalFixture { + docs: EvalDoc[]; + cases: EvalCase[]; +} + +interface EvalSummary { + cases: number; + recallCases: number; + recallAt5: number | null; + top1Cases: number; + top1Accuracy: number | null; + labeledNoResultCases: number; + /** + * Of the cases where the scorer stayed silent, how many were genuinely + * unanswerable. Only meaningful over a slice that mixes answerable and + * unanswerable cases — over a no-result-only slice it is 100% by + * construction, which is why the gate below uses `noResultRecall`. + */ + noResultPrecision: number | null; + /** + * Of the labeled no-result cases, how many the scorer correctly answered + * with nothing. This is the false-positive metric: a scorer that returns + * filler documents for an unmatched query scores 0 here. + */ + noResultRecall: number | null; + maxSelectedDocs: number; +} + +type Selector = ( + query: string, + docs: ScannedAutoMemoryDocument[], + limit?: number, +) => ScannedAutoMemoryDocument[]; + +/* ------------------------------------------------------------------------- + * Frozen pre-#8716 reference scorer. + * + * Copied verbatim from `recall.ts` at the merge-base of this branch. It is + * intentionally duplicated rather than imported: it must keep describing the + * old behaviour even after `recall.ts` changes, otherwise the "before" column + * silently tracks the "after" one and the gate stops meaning anything. + * ---------------------------------------------------------------------- */ + +const BASELINE_TYPE_KEYWORDS: Record = { + user: ['user', 'preference', 'preferences', 'background', 'role', 'terse'], + feedback: ['feedback', 'rule', 'rules', 'avoid', 'style', 'summary'], + project: ['project', 'goal', 'goals', 'incident', 'deadline', 'release'], + reference: ['reference', 'dashboard', 'ticket', 'docs', 'doc', 'link'], +}; + +function baselineNormalizeBody(body: string): string { + const trimmed = body.trim(); + return trimmed === '_No entries yet._' ? '' : trimmed; +} + +function baselineTokenize(text: string): string[] { + return Array.from( + new Set( + text + .toLowerCase() + .split(/[^a-z0-9]+/) + .map((token) => token.trim()) + .filter((token) => token.length >= 3), + ), + ); +} + +function baselineScoreDocument( + queryTokens: string[], + doc: ScannedAutoMemoryDocument, +): number { + const normalizedBody = baselineNormalizeBody(doc.body); + const haystack = [doc.type, doc.title, doc.description, normalizedBody] + .join(' ') + .toLowerCase(); + + let score = 0; + for (const token of queryTokens) { + if (haystack.includes(token)) { + score += 2; + } + if (BASELINE_TYPE_KEYWORDS[doc.type]?.includes(token)) { + score += 1; + } + } + + if (normalizedBody.length > 0) { + score += 1; + } + + return score; +} + +const baselineSelectRelevantAutoMemoryDocuments: Selector = ( + query, + docs, + limit = RECALL_AT, +) => { + const queryTokens = baselineTokenize(query); + if (queryTokens.length === 0) { + return []; + } + + return docs + .map((doc) => ({ doc, score: baselineScoreDocument(queryTokens, doc) })) + .filter(({ score }) => score > 0) + .sort((a, b) => b.score - a.score || a.doc.type.localeCompare(b.doc.type)) + .slice(0, limit) + .map(({ doc }) => doc); +}; + +/* ---------------------------------------------------------------------- */ + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function loadFixture(): EvalFixture { + const parsed: unknown = JSON.parse(readFileSync(fixtureUrl, 'utf8')); + if (!isRecord(parsed)) { + throw new Error('recall evaluation fixture must be an object'); + } + const { docs, cases } = parsed as unknown as EvalFixture; + if (!Array.isArray(docs) || !Array.isArray(cases)) { + throw new Error('recall evaluation fixture must define docs and cases'); + } + + const docIds = new Set(docs.map((doc) => doc.id)); + if (docIds.size !== docs.length) { + throw new Error('recall evaluation fixture must use unique doc IDs'); + } + + const caseIds = new Set(cases.map((testCase) => testCase.id)); + if (caseIds.size !== cases.length) { + throw new Error('recall evaluation fixture must use unique case IDs'); + } + + for (const testCase of cases) { + if (!categories.has(testCase.category)) { + throw new Error(`unknown category on case ${testCase.id}`); + } + for (const id of [ + ...testCase.relevantIds, + ...(testCase.expectedTopId ? [testCase.expectedTopId] : []), + ]) { + if (!docIds.has(id)) { + throw new Error(`case ${testCase.id} references unknown doc ${id}`); + } + } + if (testCase.expectedTopId && testCase.relevantIds.length === 0) { + throw new Error(`case ${testCase.id} expects a top doc but is no-result`); + } + } + + return { docs, cases }; +} + +/** + * Fixed mtime for every document: recency must not influence the + * deterministic scorer, so a shared value keeps the corpus order-stable. + */ +function toScannedDocs(docs: EvalDoc[]): ScannedAutoMemoryDocument[] { + return docs.map((doc) => ({ + type: doc.type, + filePath: `/memory/${doc.id}.md`, + relativePath: `${doc.id}.md`, + filename: `${doc.id}.md`, + title: doc.title, + description: doc.description, + body: doc.body, + mtimeMs: 1, + })); +} + +function docIdOf(doc: ScannedAutoMemoryDocument): string { + return doc.filename.replace(/\.md$/, ''); +} + +function evaluate( + fixture: EvalFixture, + selector: Selector, + filter: (testCase: EvalCase) => boolean = () => true, +): EvalSummary { + const scannedDocs = toScannedDocs(fixture.docs); + const cases = fixture.cases.filter(filter); + + let recallTotal = 0; + let recallCases = 0; + let top1Hits = 0; + let top1Cases = 0; + let correctNoResultPredictions = 0; + let noResultPredictions = 0; + let labeledNoResultCases = 0; + let correctlySilentNoResultCases = 0; + let maxSelectedDocs = 0; + + for (const testCase of cases) { + const selected = selector(testCase.query, scannedDocs, RECALL_AT); + const selectedIds = selected.map(docIdOf); + maxSelectedDocs = Math.max(maxSelectedDocs, selected.length); + + if (testCase.relevantIds.length > 0) { + const hits = testCase.relevantIds.filter((id) => + selectedIds.slice(0, RECALL_AT).includes(id), + ).length; + recallTotal += hits / testCase.relevantIds.length; + recallCases += 1; + } else { + labeledNoResultCases += 1; + if (selected.length === 0) { + correctlySilentNoResultCases += 1; + } + } + + if (testCase.expectedTopId) { + top1Hits += selectedIds[0] === testCase.expectedTopId ? 1 : 0; + top1Cases += 1; + } + + if (selected.length === 0) { + noResultPredictions += 1; + correctNoResultPredictions += testCase.relevantIds.length === 0 ? 1 : 0; + } + } + + // `null` means "not applicable to this slice" so an inapplicable metric + // reads as n/a rather than as a 0% failure. + return { + cases: cases.length, + recallCases, + recallAt5: recallCases === 0 ? null : recallTotal / recallCases, + top1Cases, + top1Accuracy: top1Cases === 0 ? null : top1Hits / top1Cases, + labeledNoResultCases, + noResultPrecision: + noResultPredictions === 0 + ? null + : correctNoResultPredictions / noResultPredictions, + noResultRecall: + labeledNoResultCases === 0 + ? null + : correctlySilentNoResultCases / labeledNoResultCases, + maxSelectedDocs, + }; +} + +const isEnglish = (testCase: EvalCase) => testCase.category === 'english'; +/** + * Answerable queries that share no token with their labeled document. The + * scorer requires a lexical match before it returns anything, so it cannot + * serve this slice by design — the model selector is the path that covers it. + * The slice exists to keep that cost measured and visible rather than absent + * from the corpus, and it is excluded from the quality floor below because a + * floor over cases the deterministic path is not meant to answer would only + * measure how many of them the corpus happens to contain. + */ +const isSemanticNoLexical = (testCase: EvalCase) => + testCase.category === 'semantic-no-lexical'; +/** + * Alphabetic scripts outside ASCII and CJK — Cyrillic, Greek, and accented + * Latin here. The pre-change tokenizer kept only `[a-z0-9]{3,}` runs, so + * these queries produced no tokens at all and the deterministic path was + * unconditionally silent; the shipped tokenizer keeps whole non-CJK letter + * runs instead. + */ +const isOtherScript = (testCase: EvalCase) => + testCase.category === 'other-script'; +const isCjk = (testCase: EvalCase) => + testCase.category === 'chinese' || + testCase.category === 'japanese' || + testCase.category === 'korean'; +/** Queries that mix scripts, including the full-width NFKC cases. */ +const isMixed = (testCase: EvalCase) => + testCase.category === 'mixed' || testCase.category === 'nfkc'; + +function formatPercent(value: number | null): string { + return value === null ? 'n/a' : `${(value * 100).toFixed(1)}%`; +} + +/** + * Expected Recall@5 of a scorer that ignores the query and returns five + * documents drawn uniformly at random from the corpus. Each labeled document + * has a `RECALL_AT / corpusSize` chance of being among them, so the expected + * per-case recall — and therefore the mean over any slice — is that ratio. + * + * Printed beside the measured columns because a small corpus flatters the + * headline: on a pool this size "100% Recall@5" is a much weaker statement + * than it reads, and a reader comparing designs needs the floor to calibrate + * against. It is exact rather than sampled, so the table stays deterministic. + */ +function randomBaselineRecallAt5(corpusSize: number): number { + return corpusSize === 0 ? 0 : Math.min(1, RECALL_AT / corpusSize); +} + +describe('auto-memory recall evaluation', () => { + it('loads a labeled corpus covering every required category', () => { + const fixture = loadFixture(); + expect(fixture.cases.length).toBeGreaterThanOrEqual(30); + expect(fixture.cases.length).toBeLessThanOrEqual(60); + expect(new Set(fixture.cases.map((testCase) => testCase.category))).toEqual( + categories, + ); + }); + + /** + * The floors carry deliberate headroom: this is an evaluation, not a + * behavioural contract. One known top-1 miss is expected and correct — + * `body-only-owner-approval` ("owner approval") ranks `en-owner` first + * because "owner" is a substring of "ownership" in that document's title, + * and a title match outranks the body match in `en-rollback`. Tighten the + * scorer if that ordering is ever judged wrong; do not relabel the case. + */ + it('holds the multilingual quality floor', () => { + // Excludes the semantic-no-lexical slice: those cases are labeled + // answerable but are unreachable without a lexical match, so counting + // them here would turn a deliberate design boundary into a moving floor. + // Their cost is measured by the test below instead. + const summary = evaluate( + loadFixture(), + selectRelevantAutoMemoryDocuments, + (testCase) => !isSemanticNoLexical(testCase), + ); + expect(summary.recallAt5).toBeGreaterThanOrEqual(0.9); + expect(summary.top1Accuracy).toBeGreaterThanOrEqual(0.85); + expect(summary.noResultPrecision).toBe(1); + expect(summary.noResultRecall).toBe(1); + expect(summary.maxSelectedDocs).toBeLessThanOrEqual(RECALL_AT); + }); + + /** + * Records the price of "no lexical match, no score". These queries are + * genuinely answerable — a human, and the model selector, would pick the + * labeled document — and the deterministic scorer returns nothing for all + * of them. Two consequences follow that the headline numbers do not show: + * on a tool-free turn the fast path delivers nothing here, and the + * selector-failure fallback is silent here too. + * + * If a future scorer change starts answering part of this slice, this test + * fails. That is the intended signal: raise the number, do not delete the + * cases or move them to `no-result`. + */ + it('records the deterministic path as silent on semantic-only queries', () => { + const summary = evaluate( + loadFixture(), + selectRelevantAutoMemoryDocuments, + isSemanticNoLexical, + ); + + expect(summary.recallCases).toBeGreaterThanOrEqual(3); + expect(summary.recallAt5).toBe(0); + expect(summary.maxSelectedDocs).toBe(0); + }); + + it('does not regress English Recall@5 against the pre-change scorer', () => { + const fixture = loadFixture(); + const before = evaluate( + fixture, + baselineSelectRelevantAutoMemoryDocuments, + isEnglish, + ); + const after = evaluate( + fixture, + selectRelevantAutoMemoryDocuments, + isEnglish, + ); + + expect(before.recallCases).toBeGreaterThan(0); + expect(after.recallAt5!).toBeGreaterThanOrEqual(before.recallAt5!); + expect(after.top1Accuracy!).toBeGreaterThanOrEqual(before.top1Accuracy!); + }); + + it('does not regress no-result handling against the pre-change scorer', () => { + const fixture = loadFixture(); + const before = evaluate(fixture, baselineSelectRelevantAutoMemoryDocuments); + const after = evaluate(fixture, selectRelevantAutoMemoryDocuments); + + // Measured over the whole corpus. Restricting to the no-result cases + // would make precision 100% for any scorer that ever stays silent. + expect(before.labeledNoResultCases).toBeGreaterThan(0); + expect(after.noResultRecall!).toBeGreaterThanOrEqual( + before.noResultRecall!, + ); + expect(after.noResultPrecision!).toBeGreaterThanOrEqual( + before.noResultPrecision!, + ); + }); + + it('reports the before/after rollout-gate table', () => { + const fixture = loadFixture(); + const rows = [ + ['overall', () => true], + ['english', isEnglish], + ['cjk', isCjk], + ['mixed', isMixed], + ['other-script', isOtherScript], + ['semantic-no-lexical', isSemanticNoLexical], + ] as const; + + const lines = [ + '', + 'RFC #7040 rollout gate — deterministic recall path', + '', + '| slice | metric | before | after |', + '| --- | --- | --- | --- |', + ]; + + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + lines.splice( + 2, + 0, + `corpus: ${fixture.docs.length} documents, ${fixture.cases.length} cases — a query-blind random scorer returning ${RECALL_AT} documents scores ${formatPercent(randomFloor)} Recall@5 on this pool`, + '', + ); + + for (const [label, filter] of rows) { + const before = evaluate( + fixture, + baselineSelectRelevantAutoMemoryDocuments, + filter, + ); + const after = evaluate( + fixture, + selectRelevantAutoMemoryDocuments, + filter, + ); + const metrics: Array<[string, number | null, number | null]> = [ + ['Recall@5', before.recallAt5, after.recallAt5], + ['top-1 accuracy', before.top1Accuracy, after.top1Accuracy], + [ + 'no-result precision', + before.noResultPrecision, + after.noResultPrecision, + ], + ['no-result recall', before.noResultRecall, after.noResultRecall], + ]; + for (const [metric, beforeValue, afterValue] of metrics) { + lines.push( + `| ${label} (n=${before.cases}) | ${metric} | ${formatPercent(beforeValue)} | ${formatPercent(afterValue)} |`, + ); + } + } + + console.log(lines.join('\n')); + expect(lines.length).toBeGreaterThan(5); + }); + + /** + * Guards the headline against a corpus so small that Recall@5 is nearly + * free. This is a property of the fixture, not of the scorer: shrink the + * corpus far enough and every metric approaches 100% for any design. + */ + it('keeps the corpus large enough for Recall@5 to discriminate', () => { + const fixture = loadFixture(); + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + + expect(randomFloor).toBeLessThanOrEqual(0.25); + const after = evaluate( + fixture, + selectRelevantAutoMemoryDocuments, + (testCase) => !isSemanticNoLexical(testCase), + ); + expect(after.recallAt5!).toBeGreaterThan(randomFloor * 3); + }); + + it('produces deterministic summaries', () => { + const fixture = loadFixture(); + expect(evaluate(fixture, selectRelevantAutoMemoryDocuments)).toEqual( + evaluate(fixture, selectRelevantAutoMemoryDocuments), + ); + }); +}); diff --git a/packages/core/src/memory/recall-scan-latency.test.ts b/packages/core/src/memory/recall-scan-latency.test.ts new file mode 100644 index 00000000000..7f3672f53fe --- /dev/null +++ b/packages/core/src/memory/recall-scan-latency.test.ts @@ -0,0 +1,182 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import * as fs from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import type { Config } from '../config/config.js'; +import { getAutoMemoryFilePath } from './paths.js'; +import { resolveRelevantAutoMemoryPromptForQuery } from './recall.js'; +import { selectRelevantAutoMemoryDocumentsByModel } from './relevanceSelector.js'; +import { ensureAutoMemoryScaffold } from './store.js'; + +/** + * Measures the part of the initial-turn budget nothing else measures. + * + * `recall-delivery-eval.test.ts` times the deterministic *scoring*, which is + * microseconds. That is not what decides whether the fast path delivers. The + * fast result is published from `onFastResult`, which fires only after recall + * has enumerated, read, and parsed every topic file — and this branch removed + * the 200-document cap for recall, so that scan grows with the memory tree. + * If the scan alone exceeds `INITIAL_MEMORY_RECALL_WAIT_MS`, the turn pays the + * full budget *and* delivers nothing, which is strictly worse than before. + * + * So this file measures wall-clock time from the recall call to the fast + * callback, against a real temporary memory tree, with the model selector + * mocked to hang the way a network round trip does. + * + * Timings are machine-dependent and CI is shared, so the assertions are + * deliberately loose; the printed table is the artifact worth reading. What + * is asserted is the structural claim: the fast result lands well inside the + * budget at memory-tree sizes users can plausibly reach. + */ + +vi.mock('./relevanceSelector.js', () => ({ + selectRelevantAutoMemoryDocumentsByModel: vi.fn(), +})); + +/** Mirrors INITIAL_MEMORY_RECALL_WAIT_MS in client.ts. */ +const INITIAL_BUDGET_MS = 100; +const TOPIC_COUNTS = [200, 500, 1000] as const; +const REPEATS = 5; + +let tempDir: string; +const projectRootByCount = new Map(); + +async function buildMemoryTree(topicCount: number): Promise { + const projectRoot = path.join(tempDir, `project-${topicCount}`); + await fs.mkdir(projectRoot, { recursive: true }); + await ensureAutoMemoryScaffold( + projectRoot, + new Date('2026-04-01T00:00:00.000Z'), + ); + + const referenceDir = path.dirname( + getAutoMemoryFilePath(projectRoot, 'reference/topic-0000.md'), + ); + await fs.mkdir(referenceDir, { recursive: true }); + + // Bodies are sized like real notes rather than one-liners: the scan reads + // and parses whole files, so a corpus of stubs would understate the cost. + const filler = 'Historical note about an unrelated subsystem. '.repeat(20); + await Promise.all( + Array.from({ length: topicCount }, (_, index) => + fs.writeFile( + path.join(referenceDir, `topic-${String(index).padStart(4, '0')}.md`), + [ + '---', + 'type: reference', + `name: Topic ${index}`, + `description: Reference note number ${index} about deployment history`, + '---', + '', + filler, + index === topicCount - 1 ? 'The saved codeword is SCANBENCH.' : '', + '', + ].join('\n'), + 'utf-8', + ), + ), + ); + + return projectRoot; +} + +/** Wall-clock ms from the recall call until the fast result is published. */ +async function measureTimeToFastResultMs(projectRoot: string): Promise { + let elapsed = Number.NaN; + const startedAt = performance.now(); + const recall = resolveRelevantAutoMemoryPromptForQuery( + projectRoot, + 'what is the saved scanbench codeword for deployment', + { + config: { + getSessionId: () => 'session-scan-bench', + getModel: () => 'qwen3-coder-plus', + } as Config, + onFastResult: () => { + elapsed = performance.now() - startedAt; + }, + }, + ); + + // Let the pending recall settle so it does not leak into the next sample. + await recall; + return elapsed; +} + +describe('auto-memory recall scan latency', () => { + beforeAll(async () => { + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'recall-scan-bench-')); + // The selector stands in for the network round trip: it must not settle + // before the fast callback, or the measurement would race it. Returning + // an empty selection keeps recall finishing promptly after that. + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockResolvedValue([]); + for (const topicCount of TOPIC_COUNTS) { + projectRootByCount.set(topicCount, await buildMemoryTree(topicCount)); + } + }, 120_000); + + afterAll(async () => { + if (tempDir) { + await fs.rm(tempDir, { recursive: true, force: true }); + } + }); + + it('publishes the fast result well inside the initial budget', async () => { + const rows: Array<[number, number, number]> = []; + + for (const topicCount of TOPIC_COUNTS) { + const projectRoot = projectRootByCount.get(topicCount)!; + // Warm the page cache so the first sample does not report cold I/O as + // the steady-state cost. + await measureTimeToFastResultMs(projectRoot); + + const samples: number[] = []; + for (let i = 0; i < REPEATS; i += 1) { + samples.push(await measureTimeToFastResultMs(projectRoot)); + } + samples.sort((a, b) => a - b); + const median = samples[Math.floor(samples.length / 2)]; + const worst = samples[samples.length - 1]; + rows.push([topicCount, median, worst]); + + expect(Number.isFinite(median)).toBe(true); + } + + const [smallest] = rows; + // The ordinary case must leave the rest of the budget to spare. Loose + // because CI is shared; the table is what carries the detail. + expect(smallest[0]).toBe(TOPIC_COUNTS[0]); + expect(smallest[1]).toBeLessThan(INITIAL_BUDGET_MS / 2); + + console.log( + [ + '', + 'Scan gate — time from recall start to fast result (single project scope)', + `initial budget: ${INITIAL_BUDGET_MS} ms`, + '', + `| topics | median | worst of ${REPEATS} | share of budget | fast result inside budget? |`, + '| --- | --- | --- | --- | --- |', + ...rows.map( + ([topicCount, median, worst]) => + `| ${topicCount} | ${median.toFixed(1)} ms | ${worst.toFixed(1)} ms | ${((median / INITIAL_BUDGET_MS) * 100).toFixed(1)}% | ${worst < INITIAL_BUDGET_MS ? 'yes' : 'no'} |`, + ), + '', + 'The fast result is only available once this scan completes, so this is', + 'the real precondition for the fast path delivering anything — not the', + 'scoring cost, which is microseconds.', + '', + 'Where a row reads "no", the turn spends the whole budget and still', + 'delivers nothing, which is worse than the zero-wait behaviour this', + 'branch replaced. That is why the wait ends on the fast result rather', + 'than always running to the ceiling: it removes the cost for every tree', + 'small enough to scan in time, and bounds it for the rest.', + ].join('\n'), + ); + }, 120_000); +}); diff --git a/packages/core/src/memory/recall.test.ts b/packages/core/src/memory/recall.test.ts index a65ceae2a5c..20309f494ff 100644 --- a/packages/core/src/memory/recall.test.ts +++ b/packages/core/src/memory/recall.test.ts @@ -7,24 +7,25 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { buildRelevantAutoMemoryPrompt, + MAX_FAST_RECALL_DOCS, resolveRelevantAutoMemoryPromptForQuery, selectRelevantAutoMemoryDocuments, } from './recall.js'; import type { ScannedAutoMemoryDocument } from './scan.js'; import type { Config } from '../config/config.js'; -import { scanAutoMemoryTopicDocuments } from './scan.js'; +import { scanAllAutoMemoryTopicDocuments } from './scan.js'; import { selectRelevantAutoMemoryDocumentsByModel } from './relevanceSelector.js'; vi.mock('./scan.js', async (importOriginal) => { const actual = await importOriginal(); return { ...actual, - scanAutoMemoryTopicDocuments: vi.fn(), + scanAllAutoMemoryTopicDocuments: vi.fn(), // Explicit mock — recall now unions user-level docs into the pool, so // leaving this on the real implementation would silently fall through // to the filesystem (only "works" because the path doesn't exist and // listMarkdownFiles swallows ENOENT). Defaults to an empty pool. - scanUserAutoMemoryTopicDocuments: vi.fn().mockResolvedValue([]), + scanAllUserAutoMemoryTopicDocuments: vi.fn().mockResolvedValue([]), }; }); @@ -99,6 +100,146 @@ const activeToolDocs: ScannedAutoMemoryDocument[] = [ }, ]; +function memoryDoc( + filename: string, + type: ScannedAutoMemoryDocument['type'], + title: string, + description: string, + body: string, +): ScannedAutoMemoryDocument { + return { + type, + filePath: `/tmp/${filename}`, + relativePath: filename, + filename, + title, + description, + body, + mtimeMs: 1, + }; +} + +const multilingualDocs: ScannedAutoMemoryDocument[] = [ + memoryDoc( + 'zh-deploy.md', + 'project', + '生产部署流程', + '发布检查清单', + '上线前确认监控和回滚开关。', + ), + memoryDoc( + 'zh-api.md', + 'reference', + '接口延迟排查', + 'API 性能看板', + '记录服务响应时间和告警入口。', + ), + memoryDoc( + 'ja-auth.md', + 'project', + '認証設定ガイド', + 'ユーザーログイン構成', + 'セッション設定の確認手順。', + ), + memoryDoc( + 'ja-deploy.md', + 'reference', + 'デプロイ手順', + 'リリース運用', + '本番反映前の確認事項。', + ), + memoryDoc( + 'ko-deploy.md', + 'project', + '배포 절차', + '릴리스 체크리스트', + '운영 반영 전에 모니터링을 확인한다.', + ), + memoryDoc( + 'ko-auth.md', + 'reference', + '인증 설정', + '로그인 문제 해결', + '세션 만료와 권한 구성을 확인한다.', + ), + memoryDoc( + 'en-release.md', + 'project', + 'Release process', + 'Production deployment checklist', + 'Verify monitoring before shipping.', + ), + memoryDoc( + 'en-style.md', + 'user', + 'Response preferences', + 'Concise answer style', + 'Keep explanations direct.', + ), + memoryDoc( + 'mixed-api.md', + 'reference', + 'Qwen API 限流', + 'Rate limit dashboard', + '检查 quota 和请求速率。', + ), + memoryDoc( + 'body-only.md', + 'feedback', + 'Operational notes', + 'Miscellaneous guidance', + 'Emergency rollback procedures require owner approval.', + ), + memoryDoc( + 'ja-hiragana.md', + 'user', + 'よくあるしつもん', + 'ひらがなだけでかいたあんない', + 'ひらがなのとうこにそなえたきろく。', + ), +]; + +const multilingualRecallCases: Array< + [name: string, query: string, expectedFilename: string | null] +> = [ + ['Chinese title', '生产部署', 'zh-deploy.md'], + ['Chinese description', '发布检查', 'zh-deploy.md'], + ['Chinese API title', '接口延迟', 'zh-api.md'], + ['Chinese troubleshooting', '延迟排查', 'zh-api.md'], + ['Chinese mixed ASCII', 'API 延迟', 'zh-api.md'], + ['Japanese Han title', '認証設定', 'ja-auth.md'], + ['Japanese Katakana description', 'ログイン構成', 'ja-auth.md'], + ['Japanese prolonged sound mark', 'ユーザー', 'ja-auth.md'], + ['Japanese Katakana title', 'デプロイ手順', 'ja-deploy.md'], + ['Japanese release description', 'リリース運用', 'ja-deploy.md'], + ['Japanese Hiragana-only query', 'よくあるしつもん', 'ja-hiragana.md'], + ['Korean title', '배포 절차', 'ko-deploy.md'], + ['Korean description', '릴리스 체크', 'ko-deploy.md'], + ['Korean auth title', '인증 설정', 'ko-auth.md'], + ['Korean login description', '로그인 문제', 'ko-auth.md'], + ['English title', 'release process', 'en-release.md'], + ['English description', 'production deployment', 'en-release.md'], + ['English style description', 'concise answer', 'en-style.md'], + ['English preference title', 'response preferences', 'en-style.md'], + ['Mixed-language title', 'qwen api 限流', 'mixed-api.md'], + ['Mixed-language description', 'rate limit', 'mixed-api.md'], + ['Mixed ASCII and Han', 'API 限流', 'mixed-api.md'], + ['Body-only English', 'rollback procedures', 'body-only.md'], + ['Body-only phrase', 'emergency rollback', 'body-only.md'], + ['NFKC full-width API', 'ＱＷＥＮ ＡＰＩ', 'mixed-api.md'], + [ + 'NFKC full-width English', + 'ＰＲＯＤＵＣＴＩＯＮ deployment', + 'en-release.md', + ], + ['No lexical match', 'vector database', null], + ['Single Han character', '部', null], + ['Single Japanese character', '認', null], + ['Single Hangul character', '배', null], + ['Short ASCII token', 'go', null], + ['Unrelated English terms', 'empty mismatch', null], +]; + describe('auto-memory relevant recall', () => { beforeEach(() => { vi.clearAllMocks(); @@ -118,6 +259,248 @@ describe('auto-memory relevant recall', () => { expect(selectRelevantAutoMemoryDocuments(' ', docs)).toEqual([]); }); + it.each(multilingualRecallCases)('%s', (_name, query, expectedFilename) => { + const selected = selectRelevantAutoMemoryDocuments(query, multilingualDocs); + + if (expectedFilename === null) { + expect(selected).toEqual([]); + } else { + expect(selected[0]?.filename).toBe(expectedFilename); + } + }); + + it('normalizes document text before matching', () => { + expect( + selectRelevantAutoMemoryDocuments('API', [ + memoryDoc('fw-api.md', 'reference', 'ＡＰＩ', '', ''), + ])[0]?.filename, + ).toBe('fw-api.md'); + }); + + it('weights each title and description match above a body match', () => { + const bodyMatch = memoryDoc( + 'body.md', + 'reference', + 'General notes', + 'Miscellaneous', + 'Latency dashboard troubleshooting.', + ); + const titleMatch = memoryDoc( + 'title.md', + 'reference', + 'Latency dashboard', + 'Troubleshooting reference', + 'General notes.', + ); + + expect( + selectRelevantAutoMemoryDocuments('latency dashboard', [ + bodyMatch, + titleMatch, + ])[0]?.filename, + ).toBe('title.md'); + + expect( + selectRelevantAutoMemoryDocuments('user preferences background role', [ + memoryDoc('body.md', 'user', '', '', 'Background'), + memoryDoc('title.md', 'project', 'Background', '', ''), + ])[0]?.filename, + ).toBe('title.md'); + }); + + it('applies type boosts only after a lexical match', () => { + const userDoc = memoryDoc( + 'user-cadence.md', + 'user', + 'Cadence summary', + '', + '', + ); + const projectDoc = memoryDoc( + 'project-cadence.md', + 'project', + 'Cadence summary', + '', + '', + ); + + // Both docs tie on lexical score for 'cadence'; the 'preference' token + // boosts only the user-typed doc, so it must win. Without the boost the + // docs would also tie on mtime and input order would surface the project + // doc instead. + const selected = selectRelevantAutoMemoryDocuments('cadence preference', [ + projectDoc, + userDoc, + ]); + + expect(selected[0]?.filename).toBe('user-cadence.md'); + // Type keywords alone never surface a doc without a lexical match. + expect(selectRelevantAutoMemoryDocuments('preference', [userDoc])).toEqual( + [], + ); + }); + + it('tokenizes alphabetic scripts outside ASCII and CJK', () => { + // `[a-z0-9]{3,}` produced no tokens at all for these, so the + // deterministic path was unconditionally silent — no fast result, and a + // silent selector-failure fallback. + const cyrillic = memoryDoc( + 'ru.md', + 'project', + 'Процесс развёртывания', + '', + '', + ); + const greek = memoryDoc('el.md', 'reference', 'Ρύθμιση σύνδεσης', '', ''); + const accented = memoryDoc('fr.md', 'project', 'Démarrage à froid', '', ''); + const docs = [cyrillic, greek, accented]; + + expect( + selectRelevantAutoMemoryDocuments('развёртывания', docs)[0]?.filename, + ).toBe('ru.md'); + expect( + selectRelevantAutoMemoryDocuments('σύνδεσης', docs)[0]?.filename, + ).toBe('el.md'); + expect( + selectRelevantAutoMemoryDocuments('démarrage', docs)[0]?.filename, + ).toBe('fr.md'); + }); + + it('does not let a Latin run swallow the CJK that follows it', () => { + // `\p{L}` also matches Han, so a naive alphabetic class would tokenize + // `abc漢字` as one run and stop matching either half on its own. + const latin = memoryDoc('latin.md', 'reference', 'abc', '', ''); + const han = memoryDoc('han.md', 'reference', '漢字', '', ''); + + expect( + selectRelevantAutoMemoryDocuments('abc漢字', [latin, han]).map( + (doc) => doc.filename, + ), + ).toEqual(['latin.md', 'han.md']); + }); + + it('still ignores runs shorter than three characters', () => { + const doc = memoryDoc('go.md', 'reference', 'go go go', '', ''); + + expect(selectRelevantAutoMemoryDocuments('go', [doc])).toEqual([]); + // Two Cyrillic letters are below the threshold for the same reason. + expect( + selectRelevantAutoMemoryDocuments('до', [ + memoryDoc('ru.md', 'reference', 'до свидания', '', ''), + ]), + ).toEqual([]); + }); + + it('breaks score ties by recency, not by document type', () => { + // Every type carries the same title, so the only thing separating these + // documents is the tie-break. An alphabetical type comparison orders them + // feedback < project < reference < user, which pushes user memory out of + // the two-document fast result entirely. + const withMtime = ( + doc: ScannedAutoMemoryDocument, + mtimeMs: number, + ): ScannedAutoMemoryDocument => ({ ...doc, mtimeMs }); + const docs = [ + withMtime(memoryDoc('fb.md', 'feedback', 'Deploy notes', '', ''), 10), + withMtime(memoryDoc('pr.md', 'project', 'Deploy notes', '', ''), 20), + withMtime(memoryDoc('rf.md', 'reference', 'Deploy notes', '', ''), 30), + withMtime(memoryDoc('us.md', 'user', 'Deploy notes', '', ''), 40), + ]; + + expect( + selectRelevantAutoMemoryDocuments('deploy', docs).map( + (doc) => doc.filename, + ), + ).toEqual(['us.md', 'rf.md', 'pr.md', 'fb.md']); + + // The fast path takes only the first MAX_FAST_RECALL_DOCS, so the + // tie-break decides whether user memory reaches the model at all. + expect( + selectRelevantAutoMemoryDocuments('deploy', docs) + .slice(0, MAX_FAST_RECALL_DOCS) + .map((doc) => doc.type), + ).toContain('user'); + }); + + it('falls back to input order when score and recency both tie', () => { + // Project-level documents are concatenated ahead of user-level ones in + // `resolveRelevantAutoMemoryPromptForQuery`; the stable sort is what + // preserves that precedence once every ranking key has tied. + const projectDoc = memoryDoc('p.md', 'project', 'Deploy notes', '', ''); + const userDoc = memoryDoc('u.md', 'user', 'Deploy notes', '', ''); + + expect( + selectRelevantAutoMemoryDocuments('deploy', [projectDoc, userDoc])[0] + ?.filename, + ).toBe('p.md'); + }); + + it('bounds long mixed queries while retaining their actual text edges', () => { + const codePoints = Array.from({ length: 100 }, (_, index) => + String.fromCodePoint(0x4e00 + index), + ); + const asciiTokens = Array.from( + { length: 100 }, + (_, index) => `token${String(index).padStart(3, '0')}`, + ); + const selected = selectRelevantAutoMemoryDocuments( + `${codePoints.join('')} ${asciiTokens.join(' ')}`, + [ + memoryDoc( + 'query-start.md', + 'reference', + codePoints.slice(0, 2).join(''), + '', + '', + ), + memoryDoc( + 'query-middle.md', + 'reference', + codePoints.slice(49, 51).join(''), + '', + '', + ), + memoryDoc('query-end.md', 'reference', asciiTokens.at(-1)!, '', ''), + ], + ); + + expect(selected.map((doc) => doc.filename)).toEqual([ + 'query-start.md', + 'query-end.md', + ]); + }); + + it('refreshes repeated tokens near the query tail', () => { + const tokens = Array.from( + { length: 65 }, + (_, index) => `token${String(index).padStart(3, '0')}`, + ); + const selected = selectRelevantAutoMemoryDocuments( + [...tokens.slice(0, 64), tokens[32], tokens[64]].join(' '), + [ + memoryDoc('repeated.md', 'reference', tokens[32], '', ''), + memoryDoc('stale.md', 'reference', tokens[33], '', ''), + memoryDoc('last.md', 'reference', tokens[64], '', ''), + ], + ).map((doc) => doc.filename); + + expect(selected).toContain('repeated.md'); + expect(selected).toContain('last.md'); + expect(selected).not.toContain('stale.md'); + }); + + it('does not score body text outside the surfaced prompt window', () => { + const doc = memoryDoc( + 'late-body.md', + 'reference', + 'General notes', + '', + `${'x'.repeat(1_200)}late marker`, + ); + + expect(selectRelevantAutoMemoryDocuments('late marker', [doc])).toEqual([]); + }); + it('formats selected documents as a prompt block', () => { const prompt = buildRelevantAutoMemoryPrompt([docs[0], docs[2]]); @@ -127,7 +510,7 @@ describe('auto-memory relevant recall', () => { }); it('uses model-driven selection when config is provided', async () => { - vi.mocked(scanAutoMemoryTopicDocuments).mockResolvedValue(docs); + vi.mocked(scanAllAutoMemoryTopicDocuments).mockResolvedValue(docs); vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockResolvedValue([ docs[0], ]); @@ -145,8 +528,107 @@ describe('auto-memory relevant recall', () => { expect(result.prompt).toContain('Reference Memory (reference.md)'); }); + it('bounds model candidates while retaining lexical and recent documents', async () => { + const lexicalDocs = Array.from({ length: 200 }, (_, index) => ({ + ...memoryDoc( + `lexical-${String(index).padStart(3, '0')}.md`, + 'reference', + `Overflow memory ${index}`, + 'Matching historical context', + '', + ), + mtimeMs: 0, + })); + const recentDocs = Array.from({ length: 20 }, (_, index) => ({ + ...memoryDoc( + `recent-${String(index).padStart(2, '0')}.md`, + 'reference', + `General memory ${index}`, + 'Unrelated recent context', + '', + ), + mtimeMs: 20 - index, + })); + const lexicalTarget = { + ...memoryDoc( + 'overflow-target.md', + 'reference', + 'Overflow Zephyr Marker', + 'Unique semantic target', + '', + ), + mtimeMs: 0, + }; + vi.mocked(scanAllAutoMemoryTopicDocuments).mockResolvedValue([ + ...lexicalDocs, + ...recentDocs, + lexicalTarget, + ]); + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockImplementation( + async (_config, _query, candidates) => + candidates.includes(lexicalTarget) ? [lexicalTarget] : [], + ); + + const result = await resolveRelevantAutoMemoryPromptForQuery( + '/tmp/project', + 'find the overflow zephyr marker', + { config: {} as Config }, + ); + + const modelCandidates = vi.mocked(selectRelevantAutoMemoryDocumentsByModel) + .mock.calls[0]![2]; + expect(modelCandidates).toHaveLength(200); + expect(modelCandidates[0]).toBe(lexicalTarget); + expect(modelCandidates.filter((doc) => recentDocs.includes(doc))).toEqual( + recentDocs, + ); + expect(modelCandidates[1]).toBe(recentDocs[0]); + expect(result.selectedDocs).toEqual([lexicalTarget]); + }); + + it('fills sparse lexical candidates to the model limit with recent docs', async () => { + const lexicalDocs = Array.from({ length: 3 }, (_, index) => + memoryDoc( + `lexical-${index}.md`, + 'reference', + `Sparse target ${index}`, + '', + '', + ), + ); + const recentDocs = Array.from({ length: 250 }, (_, index) => ({ + ...memoryDoc( + `recent-${String(index).padStart(3, '0')}.md`, + 'reference', + `General memory ${index}`, + '', + '', + ), + mtimeMs: 250 - index, + })); + vi.mocked(scanAllAutoMemoryTopicDocuments).mockResolvedValue([ + ...lexicalDocs, + ...recentDocs, + ]); + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockResolvedValue([]); + + await resolveRelevantAutoMemoryPromptForQuery( + '/tmp/project', + 'find the sparse target', + { config: {} as Config }, + ); + + const modelCandidates = vi.mocked(selectRelevantAutoMemoryDocumentsByModel) + .mock.calls[0]![2]; + expect(modelCandidates).toHaveLength(200); + expect(modelCandidates.filter((doc) => lexicalDocs.includes(doc))).toEqual( + lexicalDocs, + ); + expect(modelCandidates).toContain(recentDocs[100]); + }); + it('falls back to heuristic selection when model-driven selection fails', async () => { - vi.mocked(scanAutoMemoryTopicDocuments).mockResolvedValue(docs); + vi.mocked(scanAllAutoMemoryTopicDocuments).mockResolvedValue(docs); vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockRejectedValue( new Error('selector failed'), ); @@ -170,9 +652,15 @@ describe('auto-memory relevant recall', () => { }); it('keeps active tool schemas out of heuristic fallback', async () => { - vi.mocked(scanAutoMemoryTopicDocuments).mockResolvedValue(activeToolDocs); - vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockRejectedValue( - new Error('selector failed'), + vi.mocked(scanAllAutoMemoryTopicDocuments).mockResolvedValue( + activeToolDocs, + ); + let modelCandidates: ScannedAutoMemoryDocument[] = []; + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockImplementation( + async (_config, _query, candidates) => { + modelCandidates = candidates; + throw new Error('selector failed'); + }, ); const result = await resolveRelevantAutoMemoryPromptForQuery( @@ -184,6 +672,12 @@ describe('auto-memory relevant recall', () => { }, ); + expect(modelCandidates.map((doc) => doc.filePath)).not.toContain( + '/tmp/ata-tool.md', + ); + expect(modelCandidates.map((doc) => doc.filePath)).toContain( + '/tmp/ata-gotcha.md', + ); expect(result.strategy).toBe('heuristic'); expect(result.selectedDocs.map((doc) => doc.filePath)).not.toContain( '/tmp/ata-tool.md', diff --git a/packages/core/src/memory/recall.ts b/packages/core/src/memory/recall.ts index 0754e333698..625d53e8b6a 100644 --- a/packages/core/src/memory/recall.ts +++ b/packages/core/src/memory/recall.ts @@ -8,8 +8,8 @@ import * as path from 'node:path'; import type { Config } from '../config/config.js'; import { createDebugLogger } from '../utils/debugLogger.js'; import { - scanAutoMemoryTopicDocuments, - scanUserAutoMemoryTopicDocuments, + scanAllAutoMemoryTopicDocuments, + scanAllUserAutoMemoryTopicDocuments, type ScannedAutoMemoryDocument, } from './scan.js'; import { memoryAge, memoryFreshnessText } from './memoryAge.js'; @@ -17,7 +17,17 @@ import { selectRelevantAutoMemoryDocumentsByModel } from './relevanceSelector.js import { logMemoryRecall, MemoryRecallEvent } from '../telemetry/index.js'; const MAX_RELEVANT_DOCS = 5; +/** + * Upper bound on the deterministic fast result. Deliberately far below + * MAX_RELEVANT_DOCS: this path has no model judgement behind it, so it takes + * only the highest-scoring documents and leaves the remaining prompt budget + * to the model-selected result that follows. + */ +export const MAX_FAST_RECALL_DOCS = 2; const MAX_DOC_BODY_CHARS = 1_200; +const MAX_HEURISTIC_QUERY_TOKENS = 64; +const MAX_MODEL_CANDIDATE_DOCS = 200; +const RECENT_MODEL_CANDIDATE_RESERVE = 20; const debugLogger = createDebugLogger('AUTO_MEMORY_RECALL'); const ACTIVE_TOOL_USAGE_MEMORY_MARKERS = [ @@ -62,16 +72,76 @@ const TYPE_KEYWORDS: Record = { reference: ['reference', 'dashboard', 'ticket', 'docs', 'doc', 'link'], }; +/** + * Scripts tokenized as code-point bigrams because they are written without + * word separators, so a whole run is one unsegmentable token. + */ +const CJK_CLASS = + '[\\p{Script=Han}\\p{Script=Hiragana}\\p{Script_Extensions=Katakana}\\p{Script=Hangul}]'; + +/** + * One token run: either a CJK run (bigram-tokenized below) or a run of at + * least three non-CJK letters, marks, and digits (kept whole). + * + * The alphabetic alternative is `\p{L}`-based rather than `[a-z0-9]`, so + * Cyrillic, Greek, Arabic, and accented Latin produce tokens instead of + * silently producing none. It excludes CJK per character rather than relying + * on alternation order: `\p{L}` also matches Han, so a plain class would let + * a run starting in Latin swallow the CJK that follows it and turn + * `abc漢字` into one token. + * + * Scripts written without spaces and outside the CJK set (Thai, Khmer, Lao) + * still collapse into a single long token. That is no worse than the previous + * behaviour of producing nothing, but it is not segmentation. + */ +const RECALL_TOKEN_RUN = new RegExp( + `${CJK_CLASS}+|(?!${CJK_CLASS})[\\p{L}\\p{N}](?:(?!${CJK_CLASS})[\\p{L}\\p{M}\\p{N}]){2,}`, + 'gu', +); + +/** Whether a matched run is CJK, and therefore bigram-tokenized. */ +const CJK_RUN_START = new RegExp(`^${CJK_CLASS}`, 'u'); + +function normalizeRecallText(text: string): string { + return text.normalize('NFKC').toLowerCase(); +} + function tokenize(text: string): string[] { - return Array.from( - new Set( - text - .toLowerCase() - .split(/[^a-z0-9]+/) - .map((token) => token.trim()) - .filter((token) => token.length >= 3), - ), - ); + const normalized = normalizeRecallText(text); + const edgeSize = MAX_HEURISTIC_QUERY_TOKENS / 2; + const headTokens = new Set(); + const tailTokens = new Set(); + const addToken = (token: string) => { + if (headTokens.has(token)) return; + if (tailTokens.delete(token)) { + tailTokens.add(token); + return; + } + if (headTokens.size < edgeSize) { + headTokens.add(token); + return; + } + tailTokens.add(token); + if (tailTokens.size > edgeSize) { + const oldest = tailTokens.values().next(); + if (!oldest.done) tailTokens.delete(oldest.value); + } + }; + + for (const match of normalized.matchAll(RECALL_TOKEN_RUN)) { + const run = match[0]; + if (CJK_RUN_START.test(run)) { + let previous = ''; + for (const codePoint of run) { + if (previous) addToken(previous + codePoint); + previous = codePoint; + } + } else { + addToken(run); + } + } + + return [...headTokens, ...tailTokens]; } function normalizeBody(body: string): string { @@ -103,61 +173,83 @@ function toolAliases(toolName: string): string[] { ); } -function isActiveToolUsageMemory( - doc: ScannedAutoMemoryDocument, +/** + * Build the active-tool noise predicate once per recall rather than deriving + * it per document. The alias set depends only on `recentTools`, so computing + * it inside the per-document filter re-derived up to + * `MAX_RECENT_TOOL_NAMES_FOR_MEMORY` alias lists for every scanned document — + * which recall now does over an uncapped pool. + * + * Returns a predicate rather than a boolean so both filter sites share the + * hoisting; a `recentTools`-free recall short-circuits to a constant `false`. + */ +function createActiveToolUsageFilter( recentTools: readonly string[], -): boolean { +): (doc: ScannedAutoMemoryDocument) => boolean { if (recentTools.length === 0) { - return false; + return () => false; } - const haystack = [doc.title, doc.description, normalizeBody(doc.body)] - .join(' ') - .toLowerCase(); - const namesActiveTool = recentTools.some((toolName) => - toolAliases(toolName).some((alias) => haystack.includes(alias)), - ); - if (!namesActiveTool) { - return false; + const aliases = Array.from(new Set(recentTools.flatMap(toolAliases))); + if (aliases.length === 0) { + return () => false; } - if ( - DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => - haystack.includes(marker), - ) - ) { - return false; - } + return (doc) => { + const haystack = [doc.title, doc.description, normalizeBody(doc.body)] + .join(' ') + .toLowerCase(); + if (!aliases.some((alias) => haystack.includes(alias))) { + return false; + } - return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => - haystack.includes(marker), - ); + if ( + DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => + haystack.includes(marker), + ) + ) { + return false; + } + + return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => + haystack.includes(marker), + ); + }; } function scoreDocument( queryTokens: string[], doc: ScannedAutoMemoryDocument, ): number { - const normalizedBody = normalizeBody(doc.body); - const haystack = [doc.type, doc.title, doc.description, normalizedBody] - .join(' ') - .toLowerCase(); + const title = normalizeRecallText(doc.title); + const description = normalizeRecallText(doc.description); + const body = normalizeRecallText( + normalizeBody(doc.body).slice(0, MAX_DOC_BODY_CHARS), + ); - let score = 0; + let lexicalScore = 0; for (const token of queryTokens) { - if (haystack.includes(token)) { - score += 2; + if (title.includes(token)) { + lexicalScore += 4; } - if (TYPE_KEYWORDS[doc.type]?.includes(token)) { - score += 1; + if (description.includes(token)) { + lexicalScore += 3; + } + if (body.includes(token)) { + lexicalScore += 1; } } - if (normalizedBody.length > 0) { - score += 1; + if (lexicalScore === 0) { + return 0; } - return score; + const typeBoost = Math.min( + queryTokens.filter((token) => TYPE_KEYWORDS[doc.type]?.includes(token)) + .length, + 2, + ); + return lexicalScore + typeBoost; } export function selectRelevantAutoMemoryDocuments( @@ -170,12 +262,58 @@ export function selectRelevantAutoMemoryDocuments( return []; } - return docs - .map((doc) => ({ doc, score: scoreDocument(queryTokens, doc) })) - .filter(({ score }) => score > 0) - .sort((a, b) => b.score - a.score || a.doc.type.localeCompare(b.doc.type)) - .slice(0, limit) - .map(({ doc }) => doc); + return ( + docs + .map((doc) => ({ doc, score: scoreDocument(queryTokens, doc) })) + .filter(({ score }) => score > 0) + // Recency, then input order (stable sort), as the tie-breaks. NOT the + // document type: an alphabetical type comparison ranks `user` behind + // every other type, and MAX_FAST_RECALL_DOCS takes only the top two, so + // a type tie-break would systematically drop user-level memory from the + // fast result — the exact case the fast path exists to serve. + .sort((a, b) => b.score - a.score || b.doc.mtimeMs - a.doc.mtimeMs) + .slice(0, limit) + .map(({ doc }) => doc) + ); +} + +function selectModelCandidateDocuments( + query: string, + docs: ScannedAutoMemoryDocument[], + recentTools: readonly string[], + fallbackLimit: number, +): { + modelCandidates: ScannedAutoMemoryDocument[]; + fallbackDocs: ScannedAutoMemoryDocument[]; +} { + const isActiveToolNoise = createActiveToolUsageFilter(recentTools); + const eligible = docs.filter((doc) => !isActiveToolNoise(doc)); + const lexical = selectRelevantAutoMemoryDocuments( + query, + eligible, + Math.max( + MAX_MODEL_CANDIDATE_DOCS - RECENT_MODEL_CANDIDATE_RESERVE, + fallbackLimit, + ), + ); + const modelLexical = lexical.slice( + 0, + MAX_MODEL_CANDIDATE_DOCS - RECENT_MODEL_CANDIDATE_RESERVE, + ); + const selected = new Set(modelLexical.map((doc) => doc.filePath)); + const recent = eligible + .filter((doc) => !selected.has(doc.filePath)) + .sort((a, b) => b.mtimeMs - a.mtimeMs) + .slice(0, MAX_MODEL_CANDIDATE_DOCS - modelLexical.length); + const modelCandidates = modelLexical.flatMap((doc, index) => { + const recentDoc = recent[index]; + return recentDoc ? [doc, recentDoc] : [doc]; + }); + modelCandidates.push(...recent.slice(modelLexical.length)); + return { + modelCandidates, + fallbackDocs: lexical.slice(0, fallbackLimit), + }; } function truncateBody(body: string): string { @@ -221,6 +359,21 @@ export interface ResolveRelevantAutoMemoryPromptOptions { recentTools?: readonly string[]; /** When provided and aborted, suppresses logMemoryRecall telemetry for discarded results. */ abortSignal?: AbortSignal; + /** + * Invoked with a deterministic, model-free result as soon as the shared + * scan has produced candidates — before the model selector is called. + * + * The model selector is a network side query, so the full result settles in + * round-trip time. A caller with a short initial-turn budget (see + * `INITIAL_MEMORY_RECALL_WAIT_MS`) would otherwise have nothing to inject + * on a turn that makes no tool call, because there is no later safe + * delivery point on such a turn. This callback reuses the candidates the + * selector was going to score anyway, so it costs no extra scan or I/O. + * + * Fires at most once, never after `abortSignal` aborts, and never when the + * deterministic pass found nothing. + */ + onFastResult?: (result: RelevantAutoMemoryPromptResult) => void; } export interface RelevantAutoMemoryPromptResult { @@ -256,19 +409,19 @@ export async function resolveRelevantAutoMemoryPromptForQuery( // recall returns nothing at all for the rest of the session. Project- // level scan failures still bubble — they're the only mandatory side. const [projectDocs, userDocs] = await Promise.all([ - scanAutoMemoryTopicDocuments(projectRoot), - scanUserAutoMemoryTopicDocuments().catch((error: unknown) => { + scanAllAutoMemoryTopicDocuments(projectRoot), + scanAllUserAutoMemoryTopicDocuments().catch((error: unknown) => { debugLogger.warn( `User-level auto-memory scan failed; project-level recall continues: ${error instanceof Error ? error.message : String(error)}`, ); return []; }), ]); - // Project-level docs come first as a soft hint to the model-based - // selector and, in the heuristic fallback (`selectRelevantAutoMemoryDocuments`), - // as the stable-sort tie-breaker — matching the PR's "project shadows - // user" precedence. The model selector ranks by its own judgement so - // this ordering is advisory there, not enforced. + // Project-level docs come first so that, once score and mtime have tied in + // `selectRelevantAutoMemoryDocuments`, the stable sort leaves project + // memory ahead of user memory — the "project shadows user" precedence. The + // model selector ranks by its own judgement, so this ordering is advisory + // there, not enforced. const docs = filterExcludedAutoMemoryDocuments( [...projectDocs, ...userDocs], options.excludedFilePaths, @@ -295,12 +448,33 @@ export async function resolveRelevantAutoMemoryPromptForQuery( }; } + let fallbackDocs: ScannedAutoMemoryDocument[] | undefined; if (options.config) { try { + const candidates = selectModelCandidateDocuments( + query, + docs, + options.recentTools ?? [], + limit, + ); + fallbackDocs = candidates.fallbackDocs; + // Publish the deterministic candidates before blocking on the selector + // round trip. `fallbackDocs` is already lexically ranked and already has + // active-tool noise filtered out by selectModelCandidateDocuments. + if (options.onFastResult && !options.abortSignal?.aborted) { + const fastDocs = fallbackDocs.slice(0, MAX_FAST_RECALL_DOCS); + if (fastDocs.length > 0) { + options.onFastResult({ + prompt: buildRelevantAutoMemoryPrompt(fastDocs), + selectedDocs: fastDocs, + strategy: 'heuristic', + }); + } + } const selectedDocs = await selectRelevantAutoMemoryDocumentsByModel( options.config, query, - docs, + candidates.modelCandidates, limit, options.recentTools ?? [], options.abortSignal, @@ -363,14 +537,16 @@ export async function resolveRelevantAutoMemoryPromptForQuery( }; } - const heuristicDocs = docs.filter( - (doc) => !isActiveToolUsageMemory(doc, options.recentTools ?? []), - ); - const selectedDocs = selectRelevantAutoMemoryDocuments( - query, - heuristicDocs, - limit, + const isActiveToolNoise = createActiveToolUsageFilter( + options.recentTools ?? [], ); + const selectedDocs = + fallbackDocs ?? + selectRelevantAutoMemoryDocuments( + query, + docs.filter((doc) => !isActiveToolNoise(doc)), + limit, + ); const strategy: RelevantAutoMemoryPromptResult['strategy'] = selectedDocs.length > 0 ? 'heuristic' : 'none'; if (options.config && !options.abortSignal?.aborted) { diff --git a/packages/core/src/memory/relevanceSelector.test.ts b/packages/core/src/memory/relevanceSelector.test.ts index 0c82fd807d2..9d300ffff78 100644 --- a/packages/core/src/memory/relevanceSelector.test.ts +++ b/packages/core/src/memory/relevanceSelector.test.ts @@ -272,4 +272,133 @@ describe('selectRelevantAutoMemoryDocumentsByModel', () => { '/qwen/memories/user/role.md', ]); }); + + it('bounds the model manifest by UTF-8 bytes', async () => { + const largeDocs = Array.from({ length: 200 }, (_, index) => ({ + ...docs[0], + filePath: `/tmp/bounded-${index}.md`, + relativePath: `bounded-${index}.md`, + filename: `bounded-${index}.md`, + description: `${'界'.repeat(511)}😀${'x'.repeat(2_000)}`, + mtimeMs: index, + })); + vi.mocked(runSideQuery).mockImplementation(async (_config, options) => { + const error = options.validate?.({ + selected_memories: ['/tmp/bounded-199.md'], + }); + if (error) { + throw new Error(error); + } + return { selected_memories: [] }; + }); + + await expect( + selectRelevantAutoMemoryDocumentsByModel( + mockConfig, + 'semantic-only request', + largeDocs, + 5, + ), + ).rejects.toThrow('Recall selector returned unknown file path'); + + const content = vi.mocked(runSideQuery).mock.calls[0]![1].contents[0]; + const text = content?.parts?.[0]?.text ?? ''; + const manifest = text.split('Available memories:\n')[1] ?? ''; + expect(manifest).toContain('/tmp/bounded-0.md'); + expect(manifest).not.toMatch(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/); + expect(manifest).not.toContain('x'); + expect(Buffer.byteLength(manifest, 'utf8')).toBeLessThanOrEqual(25_000); + }); + + it('keeps fitting documents after an overflowing manifest entry', async () => { + const largeDocs = Array.from({ length: 16 }, (_, index) => ({ + ...docs[0], + filePath: `/tmp/large-${index}.md`, + description: '界'.repeat(512), + })); + const shortDoc = { + ...docs[1], + filePath: '/tmp/short-after-overflow.md', + description: 'short', + }; + vi.mocked(runSideQuery).mockImplementation(async (_config, options) => { + const result = { selected_memories: [shortDoc.filePath] }; + const error = options.validate?.(result); + if (error) throw new Error(error); + return result; + }); + + await expect( + selectRelevantAutoMemoryDocumentsByModel( + mockConfig, + 'check memory', + [...largeDocs, shortDoc], + 5, + ), + ).resolves.toEqual([shortDoc]); + + const content = vi.mocked(runSideQuery).mock.calls[0]![1].contents[0]; + expect(content?.parts?.[0]?.text).toContain(shortDoc.filePath); + expect(content?.parts?.[0]?.text).not.toContain('/tmp/large-15.md'); + }); + + it('keeps interleaved recent candidates inside a full manifest', async () => { + const lexicalDocs = Array.from({ length: 180 }, (_, index) => ({ + ...docs[0], + filePath: `/tmp/lexical-${index}.md`, + description: 'lexical candidate '.repeat(6), + })); + const recentDocs = Array.from({ length: 20 }, (_, index) => ({ + ...docs[1], + filePath: `/tmp/recent-${index}.md`, + description: 'recent candidate '.repeat(6), + })); + const candidates = lexicalDocs + .slice(0, recentDocs.length) + .flatMap((doc, index) => [doc, recentDocs[index]!]); + candidates.push(...lexicalDocs.slice(recentDocs.length)); + const recentTarget = recentDocs.at(-1)!; + vi.mocked(runSideQuery).mockImplementation(async (_config, options) => { + const result = { selected_memories: [recentTarget.filePath] }; + const error = options.validate?.(result); + if (error) throw new Error(error); + return result; + }); + + await expect( + selectRelevantAutoMemoryDocumentsByModel( + mockConfig, + 'common project query', + candidates, + 5, + ), + ).resolves.toEqual([recentTarget]); + }); + + it('does not let long descriptions starve lexical-first candidates', async () => { + const longDocs = Array.from({ length: 20 }, (_, index) => ({ + ...docs[0], + filePath: `/tmp/recent-${index}.md`, + description: '界'.repeat(512), + })); + const lexicalDoc = { + ...docs[1], + filePath: '/tmp/lexical-target.md', + }; + vi.mocked(runSideQuery).mockResolvedValue({ + selected_memories: [lexicalDoc.filePath], + }); + + await expect( + selectRelevantAutoMemoryDocumentsByModel( + mockConfig, + 'find the lexical target', + [lexicalDoc, ...longDocs], + 5, + ), + ).resolves.toEqual([lexicalDoc]); + + const content = vi.mocked(runSideQuery).mock.calls[0]![1].contents[0]; + expect(content?.parts?.[0]?.text).toContain(lexicalDoc.filePath); + }); }); diff --git a/packages/core/src/memory/relevanceSelector.ts b/packages/core/src/memory/relevanceSelector.ts index 01b905876b6..a89b5e7a2e9 100644 --- a/packages/core/src/memory/relevanceSelector.ts +++ b/packages/core/src/memory/relevanceSelector.ts @@ -35,6 +35,8 @@ interface RecallSelectorResponse { selected_memories: string[]; } +const MAX_MODEL_MANIFEST_BYTES = 25_000; + /** * Format memory headers as a text manifest: one line per file with * [type] filePath (ISO-timestamp): description. @@ -49,16 +51,32 @@ interface RecallSelectorResponse { * Selector sees only the header (type, path, age, description), not the * body content. */ -function formatMemoryManifest(docs: ScannedAutoMemoryDocument[]): string { - return docs - .map((doc) => { - const tag = `[${doc.type}] `; - const ts = new Date(doc.mtimeMs).toISOString(); - return doc.description - ? `- ${tag}${doc.filePath} (${ts}): ${doc.description}` - : `- ${tag}${doc.filePath} (${ts})`; - }) - .join('\n'); +function formatMemoryManifest(docs: ScannedAutoMemoryDocument[]): { + manifest: string; + includedDocs: ScannedAutoMemoryDocument[]; +} { + const lines: string[] = []; + const includedDocs: ScannedAutoMemoryDocument[] = []; + let bytes = 0; + + for (const doc of docs) { + const tag = `[${doc.type}] `; + const ts = new Date(doc.mtimeMs).toISOString(); + const line = doc.description + ? `- ${tag}${doc.filePath} (${ts}): ${doc.description.slice(0, 512).replace(/[\uD800-\uDBFF]$/, '')}` + : `- ${tag}${doc.filePath} (${ts})`; + const nextBytes = Buffer.byteLength( + `${lines.length > 0 ? '\n' : ''}${line}`, + ); + if (bytes + nextBytes > MAX_MODEL_MANIFEST_BYTES) { + continue; + } + lines.push(line); + includedDocs.push(doc); + bytes += nextBytes; + } + + return { manifest: lines.join('\n'), includedDocs }; } export async function selectRelevantAutoMemoryDocumentsByModel( @@ -73,7 +91,10 @@ export async function selectRelevantAutoMemoryDocumentsByModel( return []; } - const manifest = formatMemoryManifest(docs); + const { manifest, includedDocs } = formatMemoryManifest(docs); + if (includedDocs.length === 0) { + return []; + } // When the assistant is actively using a tool, surfacing that tool's // reference docs is noise. Pass the tool list so the selector can skip them. @@ -93,8 +114,8 @@ export async function selectRelevantAutoMemoryDocumentsByModel( }, ]; - const validFilePaths = new Set(docs.map((doc) => doc.filePath)); - const byFilePath = new Map(docs.map((doc) => [doc.filePath, doc])); + const validFilePaths = new Set(includedDocs.map((doc) => doc.filePath)); + const byFilePath = new Map(includedDocs.map((doc) => [doc.filePath, doc])); const response = await runSideQuery(config, { purpose: 'auto-memory-recall', diff --git a/packages/core/src/memory/scan.ts b/packages/core/src/memory/scan.ts index a300d7e7d3f..6ff81997d60 100644 --- a/packages/core/src/memory/scan.ts +++ b/packages/core/src/memory/scan.ts @@ -111,7 +111,7 @@ async function listMarkdownFiles(root: string): Promise { async function scanAutoMemoryDocumentsFromRoot( root: string, - opts: { deterministic?: boolean } = {}, + opts: { deterministic?: boolean; uncapped?: boolean } = {}, ): Promise { const relativePaths = await listMarkdownFiles(root); const docs = await Promise.all( @@ -159,7 +159,7 @@ async function scanAutoMemoryDocumentsFromRoot( : valid.sort( (a, b) => b.mtimeMs - a.mtimeMs || a.filename.localeCompare(b.filename), ); - return ordered.slice(0, MAX_SCANNED_MEMORY_FILES); + return opts.uncapped ? ordered : ordered.slice(0, MAX_SCANNED_MEMORY_FILES); } export async function scanAutoMemoryTopicDocuments( @@ -168,6 +168,16 @@ export async function scanAutoMemoryTopicDocuments( return scanAutoMemoryDocumentsFromRoot(getAutoMemoryRoot(projectRoot)); } +export async function scanAllAutoMemoryTopicDocuments( + projectRoot: string, +): Promise { + // ponytail: reuse the existing O(n) parsed scan; add a catalog only if + // measured topic counts make recall scanning too slow. + return scanAutoMemoryDocumentsFromRoot(getAutoMemoryRoot(projectRoot), { + uncapped: true, + }); +} + /** * Scan the user-level (cross-project) auto-memory dir. Returns an empty * array when the dir does not exist yet, so callers can union with @@ -179,6 +189,14 @@ export async function scanUserAutoMemoryTopicDocuments(): Promise< return scanAutoMemoryDocumentsFromRoot(getUserAutoMemoryRoot()); } +export async function scanAllUserAutoMemoryTopicDocuments(): Promise< + ScannedAutoMemoryDocument[] +> { + return scanAutoMemoryDocumentsFromRoot(getUserAutoMemoryRoot(), { + uncapped: true, + }); +} + /** * Scan the team (in-repo, git-tracked) auto-memory dir. Returns an empty * array when the dir does not exist yet. diff --git a/packages/core/src/telemetry/types.ts b/packages/core/src/telemetry/types.ts index a6604409046..b274cd2e310 100644 --- a/packages/core/src/telemetry/types.ts +++ b/packages/core/src/telemetry/types.ts @@ -1604,6 +1604,15 @@ export class MemoryRecallEvent implements BaseTelemetryEvent { } } +/** + * Delivery stage, orthogonal to `strategy`. `phase` says *when* a result + * reached the model — `fast` is the deterministic result injected on the + * initial turn when the model selector had not settled inside the initial + * budget, `refined` is the model-selected result. `strategy` separately says + * *how* the documents were chosen. Both dimensions are needed: a `fast` + * delivery is always `heuristic`, but a `refined` delivery may be `model` or, + * when the selector failed, `heuristic`. + */ export type MemoryRecallDeliveryPhase = 'fast' | 'refined'; export type MemoryRecallDeliveryPoint = 'initial' | 'tool_result' | 'discarded'; export type MemoryRecallDiscardReason = @@ -1612,7 +1621,9 @@ export type MemoryRecallDiscardReason = | 'reset' | 'abort' | 'shutdown' - | 'no_relevant_results'; + | 'no_relevant_results' + /** Every document the refined result selected was already delivered by the fast phase. */ + | 'already_delivered'; export class MemoryRecallDeliveryEvent implements BaseTelemetryEvent { 'event.name': 'qwen-code.memory.recall.delivery';