feat(memory): extract on automatic compaction

[YayoiNanoka]

Summary

Implements PR②-B of #1615: automatic long-term-memory extraction when Maka persists an automatic Compaction checkpoint, building on the explicit extraction pipeline merged in #2117.

The three extraction paths now have distinct roles:

• memory_remember: the user explicitly asks Maka to remember something;
• memory_extract: the Agent identifies durable information worth preserving;
• automatic Compaction extraction: a bounded safety net when context is about to be replaced by a lossy summary.

Automatic extraction is intentionally not the only content-value trigger. A short but important conversation can still use either explicit tool. Conversely, an automatic range with no valuable memory produces a no-op and each segment is capped at three model calls.

The core invariant is:

Automatic Memory extraction triggered by Compaction must not affect the primary Agent's reasoning, response, or context compression. Neither the primary Agent nor Compaction waits for Memory extraction to finish.

Automatic trigger and execution order

This PR covers automatic pre-turn Compaction, proactive mid-turn Compaction, and reactive context-overflow Compaction. Manual compactHistory remains out of scope.

1. Freeze an Event-level Memory boundary and a synchronous policy disposition before Compaction writes its checkpoint.
2. Persist that boundary inside HistoryCompactCheckpoint.
3. After persistence succeeds, fire-and-forget a lightweight recipe into the existing per-Session Memory lane.
4. Let the Agent and Compaction continue immediately.
5. Rebuild the bounded Memory request from durable RuntimeEvents in the background.

There is no async Memory capability read, history scan, message projection, or model call on the primary Compaction path.

Coverage, context, and Evidence

• Coverage is Event-granular: (committed Cursor, frozen boundary].
• RuntimeEvents remain the durable authority for Coverage and Evidence.
• Only stable user-authored text may support a persisted Memory Item.
• Assistant text is interpretation context only; Tool calls/results, Thinking, attachments, quotes, and provider-native metadata are excluded.
• The automatic request uses a portable, tool-free envelope: the latest policy-safe prior Compaction Summary, the post-Cursor User/Assistant text slice, and the Memory prompt. It does not carry the primary Agent's System Prompt, Tool Schema, active tools, or Provider options.
• All three trigger paths may request one bounded same-Session localization pass for elliptical references. Localization cannot add Evidence and cannot search across a policy_denied barrier.
• Every citation still passes the PR②-A deterministic quote, scope, sensitive-data, temporal, and admission checks.
• A range with no user Evidence advances the Cursor as a deterministic no-op without calling the model.

Request bounds and splitting

The complete auxiliary request is checked against the frozen model context window, including Messages, Memory prompt, and output reserve.

• If one range does not fit, it may be split into at most two sequential Event ranges.
• A feasible Turn boundary is preferred; an Event boundary is the fallback.
• Both segments must pass the complete-request preflight and there is no recursive splitting.
• The first segment is an independent incidental extract range with its own operation identity and Event boundary. The second segment retains the original trigger semantics and, for memory_remember, the requested Turn.
• Each segment has a maximum of three model calls.

Failure, retry, and crash recovery

Compaction checkpoints remain the durable task identities; this PR does not add a generic task table, Worker, or Lease.

• A failed Compaction range keeps its original checkpoint association.
• The next distinct trigger first retries the old range from its old Event boundary; it never substitutes or merges the new Compaction range.
• A failed split prefix is recovered from its own Event boundary rather than being incorrectly bound to the later full checkpoint.
• A pending memory_remember retry reconstructs the original requested Turn identity.
• If a range fails again on the next trigger cycle, it is recorded as discarded and the Cursor advances so later history is not blocked indefinitely.
• If the process exits after checkpoint persistence but before dispatch, a later trigger recovers the earliest unprocessed eligible checkpoint.

Policy and privacy behavior

• The backend uses a synchronous, frozen policy snapshot before checkpoint persistence; Compaction never waits for Memory policy I/O.
• Disabled, Incognito, archived, subagent, and otherwise deterministically ineligible ranges are marked policy_denied, never sent to the Memory model, and permanently skipped.
• Transient unavailable remains eligible and recoverable.
• If policy changes to denied after an eligible checkpoint was persisted, an independent durable denial marker is written before any Cursor skip. This remains durable even behind an older explicit Pending Failure.
• A denied checkpoint creates both a Summary barrier and a localization-search barrier, so later extraction cannot recover denied history through accumulated summaries or semantic lookup.
• Policy skip cannot consume an explicit memory_remember or memory_extract Pending Failure.

Storage

SQLite schema v5 adds durable Compaction policy-denial markers. The earlier v4 changes retain Compaction checkpoint association and explicit skipped / policy_denied receipt semantics. Cursor, Pending Failure, Receipt, denial marker, and Item changes preserve their transactional and idempotency guarantees; existing v3 pending rows migrate without loss.

Verification

• build and typecheck: @maka/core, @maka/storage, @maka/runtime, and @maka/runtime-host;
• Runtime focused suites: 303 passed;
• Storage focused suites: 44 passed;
• Runtime Host focused suites, including real SQLite/provider-wire coverage: 57 passed;
• Biome and git diff --check passed;
• merged current upstream/main and resolved/verified all conflicts;
• final independent adversarial review found no remaining high-confidence P0/P1 issue.

This PR intentionally excludes recall, Item updates, semantic deduplication, conflict resolution, Groups, Summaries, feedback, forgetting, and Provider Deferred Tools. Design documents remain local and are intentionally excluded from the commit and PR.

中文说明

概要

这是 #1615 长期记忆方案的 PR②-B，接续已合并的 #2117。

目前有三种提取路径：

• memory_remember：用户明确要求记住时调用；
• memory_extract：Agent 判断对话中存在值得长期保存的信息时调用；
• 自动 Compaction 提取：上下文即将被有损摘要替换时的保底机制。

因此自动触发不是唯一的“价值判断”入口。短但重要的对话仍可使用两个显式工具；自动范围即使完全没有值得保存的内容，也只会产生 no-op，并且每段最多三次模型请求。

核心不变量：

Compaction 自动触发的记忆提取不影响主 Agent 的推理、回复或上下文压缩；主 Agent 和 Compaction 都不等待记忆提取完成。

自动触发与时序

本 PR 覆盖自动 pre-turn Compaction、proactive mid-turn Compaction 和 reactive context-overflow Compaction；手动 compactHistory 暂不触发。

1. 在 Compaction checkpoint 写入前冻结 Event 级 Memory 边界和同步 policy disposition；
2. 将边界持久化到 HistoryCompactCheckpoint；
3. checkpoint 成功后 fire-and-forget 派发轻量 recipe；
4. 主 Agent 与 Compaction 立即继续；
5. 后台 per-Session Memory lane 从持久 RuntimeEvent 重建有界请求并提取。

主 Compaction 路径不等待异步 Memory gate，也不扫描历史、不构建完整 Memory 消息、不调用模型。

Coverage、上下文与 Evidence

• Coverage 以 Event 为最小粒度：(已提交 Cursor, 冻结边界]；
• RuntimeEvent 是 Coverage 和 Evidence 的持久权威；
• 只有稳定的用户文本可以支持最终持久化 Item；
• Assistant 文本只用于解释；Tool Call/Result、Thinking、附件、Quote 和 Provider 私有元数据全部排除；
• 自动请求使用 portable、tool-free envelope：最近一次安全的旧 Compaction Summary、Cursor 后的 User/Assistant 增量文本，以及 Memory Prompt；不携带主 Agent 的 System Prompt、Tool Schema、active tools 或 Provider options；
• 三种触发都允许一次有界的同 Session 定位，用于理解“对，就按这个方案”一类省略表达；定位不能扩大 Evidence，也不能跨越 policy_denied 隐私屏障；
• 所有引用继续经过 PR②-A 的 quote、scope、敏感信息、时间字段和 admission 确定性校验；
• 范围内没有用户 Evidence 时不调用模型，直接 no-op 推进 Cursor。

请求上限与两段切分

每次都会按完整辅助请求检查容量，包括 Messages、Memory Prompt 和输出预留。

• 单段放不下时最多切成两个顺序执行的 Event 区间；
• 优先选择能让两段都放下的 Turn 边界，否则使用 Event 边界；
• 两段都必须通过完整请求预检，不递归继续切分；
• 第一段是拥有独立 operation identity 和 Event 边界的 incidental extract；第二段保留原始触发语义，memory_remember 的请求 Turn 也保留在第二段；
• 每段最多三次模型请求。

失败、重试与崩溃恢复

• Compaction 失败范围保留原 checkpoint 关联；
• 下一次不同触发先按旧 Event 边界恢复旧范围，不能用新范围替代或合并；
• 切分后的第一段失败时按自己的 Event 边界恢复，不错误绑定尚未到达的完整 checkpoint；
• memory_remember Pending Failure 会恢复原请求 Turn 身份；
• 同一范围在下一个触发周期再次失败时记录 discarded 并推进 Cursor，避免永久阻塞；
• 进程在 checkpoint 持久化后、后台派发前崩溃时，后续触发恢复 Cursor 之后最早的未处理 eligible checkpoint。

Policy 与隐私语义

• checkpoint 写入前使用 Backend 冻结的同步 policy snapshot，Compaction 不等待 Memory policy I/O；
• Disabled、Incognito、归档、subagent 等确定性拒绝写入 policy_denied，不调用 Memory 模型，并永久跳过该范围；
• 瞬时 unavailable 保持 eligible，之后可以恢复；
• eligible checkpoint 在后台执行前变成拒绝时，先独立持久化 denial marker，再尝试推进 Cursor；即使前方存在旧的显式 Pending Failure，拒绝事实也不会丢失；
• denied checkpoint 同时构成 Summary 屏障和定位搜索屏障，后续提取不能通过累计 Summary 或语义搜索恢复禁用期历史；
• policy skip 不能吞掉显式 memory_remember 或 memory_extract 的 Pending Failure。

Storage 与验证

SQLite schema v5 增加独立的 Compaction policy-denial marker；v4 已加入 Pending Failure 的 checkpoint 关联和 skipped / policy_denied Receipt 语义。v3 既有 Pending Failure 可以无损迁移。

验证结果：

• 四个相关 workspace build/typecheck 通过；
• Runtime 聚焦测试 303 passed；
• Storage 聚焦测试 44 passed；
• Runtime Host 聚焦测试 57 passed，包含真实 SQLite 与 Provider wire；
• Biome 与 git diff --check 通过；
• 已合并最新 upstream/main、解决冲突并重新验证；
• 最终独立对抗审查未发现剩余高置信 P0/P1。

本 PR 不实现召回、Item 更新、语义去重、冲突消解、Group、Summary、反馈、遗忘或 Provider Deferred Tools。设计文档按约定仅在本地同步，不进入 Commit 或 PR。

Part of #1615. Follows #2117.

[likun666661]

I reviewed this PR against the #1615 design trail and the explicit extraction slice in #2117. The safety work here is strong: durable checkpoint identity, user-text-only evidence, repeated policy checks, fail-open dispatch, per-session serialization, and atomic Cursor/Receipt/Item settlement are all good choices.

I do think this needs changes before merge, for two architectural reasons and two concrete correctness/privacy issues.

1. Compaction is a useful scheduling point, but not the semantic definition of automatic memory

Compaction correlates with token volume, not future memory value. A short, high-value session may never compact, while a long noisy session may compact repeatedly. Since RuntimeEvents remain durable, Compaction is not a last-chance evidence boundary either; it is a control-plane scheduling choice.

That is fine if this slice is described narrowly as a Compaction-triggered coverage safety net. It should not be treated as completing automatic memory coverage or the #1615 user outcome.

2. Coverage boundary and interpretation-context boundary are currently conflated

The accepted #1615 direction says to process unprocessed RuntimeEvents covered by that Compaction. This implementation instead freezes the last RuntimeEvent before compression:

• pre-turn: lastNonCompactRuntimeEvent(priorRuntimeContext)
• mid-turn: orderedEvents.at(-1)

The Memory Cursor is then advanced through that boundary, including retained tail Events that were not actually compacted.

Those are two different boundaries:

• coverageThrough: the last Event actually being compacted/consumed by Memory
• contextThrough: the later Event the extractor may inspect for interpretation

Using one boundary for both can process recent retained-tail assertions too early. If later conversation makes their durable meaning clear, those Events are already behind the Cursor and cannot be cited as current incidental-extraction evidence.

I suggest storing both boundaries, advancing the Cursor only through the actual Compaction coverage, while optionally exposing a bounded retained tail as context.

3. The model request is not actually bounded

The Evidence JSON is bounded, but the total provider request is not. Automatic recovery rebuilds every model-visible RuntimeEvent from Session start through the checkpoint boundary:

https://github.com/maka-agent/maka-agent/blob/db196f88f7573ac272fe531be4618f3285381d91/packages/runtime/src/memory-extraction.ts#L168-L285

The host then sends all of snapshot.sourceMessages, plus the Memory prompt and Tool Schema, to the same model without an input-capacity check:

https://github.com/maka-agent/maka-agent/blob/db196f88f7573ac272fe531be4618f3285381d91/packages/runtime-host/src/server/execution-model-authority.ts#L127-L166

This is contradictory at the exact target condition: the primary request has just required Compaction because its context is too large, then Memory reconstructs the pre-Compaction prefix and sends it again with additional prompt overhead.

Repeated Compactions also repeatedly send the full historical prefix even when the Memory Cursor has advanced, so cumulative cost can grow superlinearly.

The full auxiliary request needs a hard token/input bound: process the unprocessed coverage in chunks, include only a bounded context neighborhood, and run the same provider-capacity check before generation.

4. Crash recovery is conditional on another trigger

The checkpoint is a durable task identity, but there is no startup recovery scan for Memory Compaction checkpoints. If the process exits after checkpoint persistence and the Session never triggers Memory again, extraction never runs.

The PR body accurately says “a later trigger” recovers it, so the current guarantee is next-trigger recovery, not eventual recovery. Either:

• add startup/outbox recovery for eligible unsettled checkpoints; or
• explicitly define and accept best-effort/next-trigger liveness.

A durable identity alone does not provide eventual execution.

Concrete implementation blockers

1. Reused Tool Call IDs mispair results

buildMemoryCompactionSourceContext stores Tool Results in a Map<toolCallId, result>:

https://github.com/maka-agent/maka-agent/blob/db196f88f7573ac272fe531be4618f3285381d91/packages/runtime/src/memory-extraction.ts#L196-L221

If two durable occurrences reuse the same Tool Call ID, the later result overwrites the earlier one and is attached to the first call. The repository already has replay tests establishing that reused IDs must be handled by durable occurrence, not ID alone. Pairing should be occurrence/step/order aware.

2. Attachment references in Tool Call input are not removed

Tool Result output is sanitized, but Tool Call input is serialized directly:

https://github.com/maka-agent/maka-agent/blob/db196f88f7573ac272fe531be4618f3285381d91/packages/runtime/src/memory-extraction.ts#L288-L324

An input such as maka://runtime/attachments/private-artifact enters the Memory model context, contrary to the PR's “attachment references excluded” invariant. The same recursive redaction/omission policy needs to cover Tool Call inputs.

3. Current CI is red

test_runtime_host currently reports 786 passed / 1 failed. The production Host integration expected 9 provider requests and observed 10 after the background Memory request:

https://github.com/maka-agent/maka-agent/actions/runs/31307841561/job/93230928279

This looks like a test isolation/accounting race introduced by the new auxiliary request, but it still needs to be fixed before merge.

Recommendation

I would keep the overall architecture, but request these changes:

1. separate Compaction coverage from extraction interpretation context;
2. bound the complete auxiliary model request, not only Evidence JSON/model-call count;
3. decide and encode the liveness guarantee (startup recovery vs explicitly conditional next-trigger recovery);
4. fix occurrence-based Tool Result pairing and Tool Call attachment sanitization;
5. restore green CI.

The missing recall/quality evaluation does not have to block this infrastructure slice if the scope stays narrow, but it does mean this PR proves pipeline safety—not yet the #1615 user-level memory capability.

[likun666661]

Re-reviewed the current head after the bounded-extraction follow-up.

The earlier implementation blockers are addressed in a coherent way:

• automatic Compaction extraction now rebuilds a portable, tool-free User/Assistant text slice from durable RuntimeEvents;
• the complete auxiliary request is preflighted against the frozen model context window, with at most one two-way split and a bounded model-call budget per segment;
• Tool calls/results and attachment references are excluded rather than partially sanitized;
• checkpoint identity, per-Session serialization, Cursor/Receipt/Item settlement, retry/discard behavior, and durable policy-denial barriers provide a strong failure and privacy story;
• the Host integration now accounts for the asynchronous auxiliary request.

The remaining tradeoffs are explicit and acceptable for this PR's scope: Compaction is a coverage safety net rather than the sole semantic value trigger, and crash recovery is next-trigger recovery rather than startup-driven eventual execution. I do not consider those merge blockers for this infrastructure slice.

Overall, this now looks like a mature asynchronous Memory extraction foundation. Extraction quality, recall, semantic deduplication, updates/conflict handling, feedback, and forgetting can be evaluated and developed independently without expanding this already substantial PR.

LGTM from me.