fix(memory): improve recall reliability and candidate coverage

docs/design/2026-05-15-async-memory-recall-design.md

@@ -5,6 +5,14 @@
 **Related issues:** #3761, #3759
 **Related PRs:** #3814, #3866
 
+> **Updated 2026-08-08:** The UserQuery consume point now waits for at most
+> 100 ms. If the recall settles inside that budget it is delivered initially;
+> if it does not, a deterministic fast result is delivered instead of nothing,
+> and the pending model-selected result is preserved for ToolResult delivery
+> with the fast-delivered documents excluded. The zero-wait UserQuery
+> statements and behavior-table rows below describe the original design and are
+> superseded by `2026-08-08-native-memory-recall-reliability.md`.
+
 ---
 
 ## Problem

docs/design/2026-08-08-native-memory-recall-reliability.md

@@ -0,0 +1,140 @@
+# Native Memory Recall Reliability
+
+## Problem
+
+Managed-memory recall starts asynchronously for each user query. The initial
+request originally performed a zero-wait consume, so a useful selector result
+could miss the first prompt. If the turn has no tool call, that result has no
+later safe delivery point and is discarded.
+
+A fixed 100 ms initial budget was the first attempt at a fix. Measurement
+showed it is not sufficient on its own. Recall awaits the model selector
+whenever a `Config` is present — the normal case — and that selector is a
+network side query whose abort ceiling is 30 s. The budget is therefore
+dominated by round-trip time, not by the incidental scheduler timing it was
+sized for, so it expires on the common path. Delivery then falls through to the
+ToolResult point, which a tool-free turn never reaches. Tool-free turns are
+exactly the ones where user-level memory matters most: short questions answered
+from context rather than from the repository.
+
+The model selector remains the normal precision gate. Its failure fallback had
+two independent correctness problems: it tokenized only ASCII text and gave
+every non-empty document a positive score even without a lexical match.
+
+## Decision
+
+Keep a single recall lifecycle and model-primary selection. Add one
+deterministic delivery stage in front of it — not the two-stage shared-scan
+Fast/Refined architecture originally proposed in RFC #7040.
+
+- Give user-query recall a fixed 100 ms initial wait budget.
+- Deliver a result that settles inside the budget in the initial prompt.
+- If the budget expires and the deterministic candidate pass found relevant
+  documents, deliver that bounded result instead of nothing.
+  `selectModelCandidateDocuments` already computes lexically ranked candidates
+  in order to build the model manifest, so the fast result reuses them and
+  costs no extra scan or I/O. It is capped at two documents
+  (`MAX_FAST_RECALL_DOCS`), well below the five-document prompt limit, because
+  it carries no model judgement.
+- Leave recall pending after a fast delivery so the model-selected result still
+  lands at the existing same-query ToolResult delivery point.
+- Exclude documents the fast phase already delivered from that later delivery,
+  rebuilding the prompt from what remains. Both results come from one scan, so
+  the selector never saw the fast documents as excluded and can legitimately
+  re-select them. When every selected document was already delivered, record
+  `already_delivered`; when the selector returned no documents at all, record
+  `no_relevant_results`.
+- Do not abort recall merely because the initial budget expires.
+- Preserve the existing cancellation and exactly-once terminal telemetry paths.
+  A cancelled turn delivers no fast result.
+
+The 100 ms budget stays internal, per RFC #7040's direction of a small fixed
+internal budget determined by benchmark rather than exposed as public
+configuration; telemetry can show whether a later change is justified.
+
+### Why not the original Fast/Refined architecture
+
+RFC #7040 specified two results produced from one shared scan, with the refined
+pass excluding already-delivered fast documents and filling up to a combined
+five-document limit. The delivery guarantee that design existed to provide is
+worth having; its machinery is not. A second selection pathway needs its own
+scan plumbing, its own budget accounting, and cross-phase document bookkeeping —
+and that bookkeeping is the source of the duplicate-injection class of bug the
+RFC itself warned about. Reusing the candidates the selector was already going
+to score gets the same guarantee from one added callback and one exclusion set.
+
+### Telemetry: `phase` and `strategy` are orthogonal
+
+`phase` is the **delivery stage**: `fast` for a deterministic result injected
+at budget expiry, `refined` for the model-selected result. `strategy` is the
+**selection method**: `none`, `heuristic`, or `model`. They are not redundant
+and neither subsumes the other. A `fast` delivery is always `heuristic`, but a
+`refined` delivery is `model` normally and `heuristic` when the selector failed
+and the fallback ran. Reading delivery-stage behaviour off `strategy` alone
+would silently merge "the deterministic result arrived first" with "the model
+selector broke".
+
+Improve the deterministic scorer, which now serves both the fast path and the
+selector-failure fallback:
+
+- normalize query and document text with Unicode NFKC;
+- keep ASCII alphanumeric tokens of at least three characters;
+- generate Unicode code-point bigrams for Han, Hiragana, Katakana, and Hangul
+  runs;
+- ignore isolated CJK characters;
+- bound fallback query tokens while retaining tokens from both ends;
+- score only the body window that can be surfaced in the prompt;
+- require a title, description, or body lexical match before applying a type
+  boost;
+- weight each title and description token match above a body token match.
+
+## Non-goals
+
+- No second scan, second selector, or separate fast/refined budget accounting.
+- No public recall timing or retrieval-mode setting.
+- No new tokenizer or retrieval dependency.
+- No change to memory writes, scopes, extraction, DREAM, forget, or compaction.
+- No removal of the shared scanner's 200-document cap for non-recall callers.
+  Recall alone uses the bounded broad-candidate design documented in
+  `2026-08-09-bounded-memory-recall-candidates.md`.
+
+## Verification
+
+Recall quality is measured in `packages/core/src/memory/recall-eval.test.ts`
+against a 45-case labeled corpus, scored both by the shipped scorer and by a
+frozen copy of the pre-change one so "no regression" is reproducible rather
+than asserted. Delivery is measured separately in `recall-delivery-eval.test.ts`,
+because a correct selection that never reaches the model is worth nothing.
+
+- Recall settling inside the budget is delivered initially.
+- A budget expiry with deterministic candidates delivers that bounded result
+  rather than nothing, and leaves recall alive for later ToolResult delivery.
+- The later delivery never repeats a document the fast phase already sent.
+- Cancellation ends the bounded wait and prevents stale delivery.
+- A fast result never crosses a query boundary.
+- No-result queries stay silent under both designs.
+- A labeled set covers Chinese, English, Japanese, Korean, mixed text,
+  NFKC normalization, body-only matches, and no-result queries.
+- Long CJK queries keep bounded scoring work and preserve both query ends.
+- Existing active-tool noise filtering remains unchanged on the deterministic
+  candidate path. The model-selector failure fallback still triggers and
+  returns at most five documents; its scoring quality intentionally improves
+  per the scorer changes above (measured by the frozen-scorer comparison in
+  `recall-eval.test.ts`).
+
+### Known limitations
+
+- The delivery evaluation models selector latency rather than measuring it; a
+  network round trip cannot be timed in a unit test. Results are reported per
+  latency scenario, and the structural claim — that a selector slower than the
+  budget leaves a tool-free turn with no delivery point under the single-path
+  design — holds for every scenario above the budget.
+- The fast result has no model judgement behind it. It is capped at two
+  documents to bound the cost of being wrong, but on a tool-free turn where the
+  selector never lands, a mis-ranked fast document is what the model sees.
+- Scoring is substring-based, so a query token can match inside a longer word
+  ("owner" inside "ownership"). The evaluation corpus records one such case
+  rather than hiding it.
+- Recall can see older documents outside the shared 200-document scanner cap,
+  but non-recall callers, including Forget, keep the existing capped scanner.
+  A broader manageability pass is separate from this recall-only change.

docs/design/2026-08-09-bounded-memory-recall-candidates.md

@@ -0,0 +1,68 @@
+# Bounded Memory Recall Candidates
+
+## Problem
+
+The project and user memory scanners enumerate, read, and parse every topic,
+then return only the 200 most recent documents. Recall uses those shared scanner
+APIs, so an older relevant document outside either 200-document window cannot
+reach the heuristic or model selector even though the expensive scan work has
+already happened.
+
+The same capped APIs are also used by Forget, Indexer, Status, and Extraction.
+Removing their limit globally would widen unrelated behavior.
+
+## Decision
+
+Keep the existing scanner APIs and their 200-document limit unchanged. Add
+explicit all-topic variants used only by recall.
+
+Recall ranks the combined project and user pool before model selection:
+
+- retain up to 180 documents with a lexical match using the existing scorer;
+- fill the remaining candidate slots by recency, preserving at least 20 recent
+  opportunities when enough lexical matches exist;
+- interleave recent opportunities with lexical candidates so the manifest byte
+  budget cannot systematically exclude the entire recent reserve;
+- send at most 200 candidates to the model selector;
+- append manifest entries only while their cumulative UTF-8 size remains at or
+  below 25,000 bytes;
+- validate selector output only against documents actually present in that
+  bounded manifest.
+
+The heuristic fallback continues to score the complete recall pool and still
+returns at most five documents. Existing body and prompt limits remain
+unchanged.
+
+Forget, Indexer, Status, and Extraction keep the capped scanner. That preserves
+their current behavior but means an older document can become recallable before
+it becomes manageable by those non-recall flows.
+
+## Failure and compatibility boundaries
+
+Project scanning remains required. User scanning remains best-effort. Invalid
+or unreadable files keep the existing skip behavior. Empty candidate manifests
+return no model selection rather than sending an unbounded request.
+
+There is no public setting, persistent index, new dependency, provider API, or
+second selection pathway. Each recall enumerates, reads, and parses the full
+project and user memory trees once, then performs O(n) local ranking and
+active-tool filtering over the parsed documents. The deterministic fast path
+described in `2026-08-08-native-memory-recall-reliability.md` reuses the
+candidates produced by that single pass, so it adds no scan, no ranking work,
+and no state machine — only an earlier delivery point for results already
+computed. The model candidate count and manifest are
+bounded, but the local I/O and filtering work grow with the memory tree; a
+persistent catalog requires separate measurement and evidence.
+
+## Verification
+
+- A deliberately old relevant topic beyond the regular 200-document result is
+  recalled from a real temporary memory tree.
+- The regular scanner still returns 200 documents and omits that topic.
+- The model candidate set contains the lexical target and recent reserve while
+  remaining at 200 documents.
+- A manifest built from large multibyte descriptions stays within 25,000 UTF-8
+  bytes.
+- A real temporary memory-tree integration test verifies overflow-topic recall;
+  client tests independently verify bounded initial waiting and later
+  ToolResult delivery.

docs/design/auto-memory/memory-system.md

@@ -350,35 +350,37 @@ flowchart TD
 
 ```mermaid
 flowchart TD
-    A[resolveRelevantAutoMemoryPromptForQuery] --> B[scanAutoMemoryTopicDocuments\n扫描所有主题文件]
-    B --> C[filterExcludedAutoMemoryDocuments\n过滤本轮已写入的文件]
+    A[resolveRelevantAutoMemoryPromptForQuery] --> B[scanAllAutoMemoryTopicDocuments +\nscanAllUserAutoMemoryTopicDocuments\n扫描项目级与用户级全部主题文件]
+    B --> C[filterExcludedAutoMemoryDocuments\n合并作用域并过滤排除列表中的文件]
     C --> D{query 为空\n或 docs 为空\n或 limit <= 0?}
     D -- 是 --> E[返回空 prompt\nstrategy: none]
     D -- 否 --> F{是否配置了 Config?}
-    F -- 是 --> G[selectRelevantAutoMemoryDocumentsByModel\n发起 side query 请求模型选择]
-    G --> H{模型返回结果?}
-    H -- 有文档 --> I[strategy: model]
-    H -- 无文档 --> J[strategy: none\n仍然返回空]
-    G -- "失败/异常" --> K[回退到启发式选择]
-    F -- 否 --> K
-    K --> L[tokenize query\n提取 ≥3 字符的 token]
-    L --> M[scoreDocument 打分\n关键词匹配 +2 / 类型关键词 +1 / 有内容 +1]
-    M --> N[过滤 score=0 的文档\n按分数降序排列，取 Top 5]
-    N --> O{有得分文档?}
-    O -- 是 --> P[strategy: heuristic]
-    O -- 否 --> J
-    I --> Q[buildRelevantAutoMemoryPrompt\n构建 Relevant Memory 区块]
-    P --> Q
-    Q --> R[返回注入主系统提示的 prompt 片段]
+    F -- 是 --> G[selectModelCandidateDocuments\n词法候选 + recent reserve\n最多 200 篇且交错排列]
+    G --> H[selectRelevantAutoMemoryDocumentsByModel\n构建最多 25 KB manifest\n发起 side query 请求模型选择]
+    H --> I{模型返回结果?}
+    I -- 有文档 --> J[strategy: model]
+    I -- 无文档 --> K[strategy: none\n仍然返回空]
+    H -- "失败/异常" --> L[复用已计算的启发式排序]
+    F -- 否 --> M[tokenize query\nNFKC + ASCII token + CJK bigram\n最多 64 个 token]
+    M --> N[scoreDocument 打分\ntitle +4 / description +3 / body +1\n词法命中后类型加成最多 +2]
+    N --> O[过滤 score=0 的文档\n按分数降序排列，取 Top 5]
+    L --> O
+    O --> P{有得分文档?}
+    P -- 是 --> Q[strategy: heuristic]
+    P -- 否 --> K
+    J --> R[buildRelevantAutoMemoryPrompt\n构建 Relevant Memory 区块]
+    Q --> R
+    R --> S[返回注入主系统提示的 prompt 片段]
 ```
 
 **评分规则（启发式）**：
 
-| 条件                             | 加分             |
-| -------------------------------- | ---------------- |
-| query token 出现在文档内容中     | +2（每个 token） |
-| query token 是该类型的特征关键词 | +1（每个 token） |
-| 文档 body 非空                   | +1               |
+| 条件                                       | 加分                |
+| ------------------------------------------ | ------------------- |
+| query token 出现在 title                   | +4（每个 token）    |
+| query token 出现在 description             | +3（每个 token）    |
+| query token 出现在 body 前 1200 字符       | +1（每个 token）    |
+| 至少一次词法命中后，token 是类型特征关键词 | +1，整篇文档最多 +2 |
 
 **每种类型的特征关键词**：
 
@@ -394,6 +396,43 @@ flowchart TD
 - 超出截断时追加提示："NOTE: Relevant memory truncated for prompt budget."
 - 包含文档的新鲜度信息（基于文件 mtime）
 
+### 投递时机（Delivery）
+
+"选中了 Memory" 不等于 "主模型看到了 Memory"。Recall 在 UserQuery 到达时异步启动，
+投递发生在两个时机：
+
+```mermaid
+flowchart TD
+    A[UserQuery 到达\n启动 Recall Prefetch] --> B{100 ms 内\nRecall 是否完成?\nINITIAL_MEMORY_RECALL_WAIT_MS}
+    B -- 是 --> C{选中结果非空?}
+    C -- 是 --> C1[注入首轮 Prompt\nphase: refined]
+    C -- 否 --> C0[丢弃\nno_relevant_results]
+    B -- 否 --> D{是否有确定性\nFast 结果?}
+    D -- 是 --> E[注入首轮 Prompt\nphase: fast\n最多 2 篇 MAX_FAST_RECALL_DOCS]
+    D -- 否 --> F[首轮不注入]
+    E --> G[Recall 继续运行]
+    F --> G
+    G --> H{本轮是否有\nToolResult?}
+    H -- 是 --> I{Recall 是否已完成?}
+    I -- 是 --> J[排除 Fast 已投递文档\n按剩余文档重建 Prompt]
+    I -- 否 --> M[丢弃\nno_safe_delivery_point]
+    J --> J1{还有剩余文档?}
+    J1 -- 是 --> K[注入 ToolResult\nphase: refined]
+    J1 -- 否 --> L{Recall 选中了文档?}
+    L -- 是 --> L1[丢弃\nalready_delivered]
+    L -- 否 --> L2[丢弃\nno_relevant_results]
+    H -- 否 --> M
+```
+
+**为什么需要 Fast 阶段**：当存在 Config 时 Recall 会等待 Model Selector，
+而它是一次网络 Side Query（中止上限 30 秒），因此 100 ms 预算通常会超时。
+若没有 Fast 阶段，**没有工具调用的轮次将完全拿不到 Memory**——而这正是
+用户级 Memory 最重要的场景。Fast 结果复用 `selectModelCandidateDocuments`
+为 Model Manifest 已经算好的候选，不产生额外扫描或 I/O。
+
+**去重**：两个阶段来自同一次扫描，Model Selector 并未把 Fast 文档视为已排除，
+因此 ToolResult 投递前必须过滤掉 Fast 已投递的 `filePath` 并重建 Prompt。
+
 ---
 
 ## Forget — 遗忘
@@ -489,6 +528,25 @@ flowchart TD
 | `strategy`      | `'none'` \| `'heuristic'` \| `'model'` | 选择策略         |
 | `duration_ms`   | number                                 | 总耗时（毫秒）   |
 
+### Recall Delivery 遥测
+
+记录选中的 Memory 是否真的送达主模型（Selection 事件无法回答这个问题）。
+
+| 字段             | 类型                                                                                                                                      | 说明                                                       |
+| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
+| `phase`          | `'fast'` \| `'refined'`                                                                                                                   | **投递阶段**                                               |
+| `delivery_point` | `'initial'` \| `'tool_result'` \| `'discarded'`                                                                                           | 投递位置                                                   |
+| `discard_reason` | `'no_safe_delivery_point'` \| `'new_query'` \| `'reset'` \| `'abort'` \| `'shutdown'` \| `'no_relevant_results'` \| `'already_delivered'` | 丢弃原因                                                   |
+| `strategy`       | `'none'` \| `'heuristic'` \| `'model'`                                                                                                    | **选择方式**                                               |
+| `docs_selected`  | number                                                                                                                                    | 结果文档数（投递事件为实际投递数；discarded 事件为选中数） |
+| `latency_ms`     | number                                                                                                                                    | 自发起的耗时                                               |
+
+> **`phase` 与 `strategy` 正交，互不替代。** `phase` 描述**何时**送达：`fast` 是预算
+> 超时后注入的确定性结果，`refined` 是 Model Selector 选出的结果。`strategy` 描述
+> **如何**选出。`fast` 投递必然是 `heuristic`；`refined` 投递常规为 `model`，
+> 在 Selector 失败走 Fallback 时为 `heuristic`。仅凭 `strategy` 判断阶段，
+> 会把"确定性结果先到"与"Selector 故障"混为一谈。
+
 ---
 
 ## 相关源文件索引