Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
e26f69c
feat(omni): collect multimodal media memory at recognition and policy…
LaZzyMan Aug 11, 2026
2fc0e7a
fix(omni): label keyframe and clip artifacts with their roles
LaZzyMan Aug 11, 2026
3becffd
feat(omni): add session resource registry and memory recall service
LaZzyMan Aug 11, 2026
eec3ae6
feat(omni): mount session media resource registry on delivery
LaZzyMan Aug 11, 2026
c6f69a5
feat(omni): add active-mode media memory recall tool and session hand…
LaZzyMan Aug 11, 2026
051ddf8
feat(omni): resolve resourceId to inputPath at the media-policy call …
LaZzyMan Aug 11, 2026
22cc1f1
feat(omni): passive sideQuery media-memory recall before the main req…
LaZzyMan Aug 11, 2026
fce4230
fix(omni): propagate the recall tool display name to every drift-guar…
LaZzyMan Aug 12, 2026
1c9fcff
fix(omni): reject unknown omni.memory root keys, stop advising on del…
LaZzyMan Aug 12, 2026
ff801db
fix(omni): bump clip/keyframe descriptor versions alongside their rol…
LaZzyMan Aug 12, 2026
3d7da79
fix(omni): prune malformed memory record values instead of blacking o…
LaZzyMan Aug 12, 2026
01eb9be
fix(omni): give every file its own policy execution and derivative rows
LaZzyMan Aug 12, 2026
132b719
fix(omni): anchor tool-result media to the object store, not its stag…
LaZzyMan Aug 12, 2026
0190640
feat(omni): commit model and client policy successes through the same…
LaZzyMan Aug 12, 2026
fad9900
feat(omni): reuse recorded policy outputs instead of re-running the tool
LaZzyMan Aug 12, 2026
9384faf
docs(omni): reconcile the memory design with the S5 issue acceptance …
LaZzyMan Aug 12, 2026
b647b3c
fix(omni): reuse text products too, and fail on unknown omni settings…
LaZzyMan Aug 12, 2026
3fb2c4c
fix(omni): stop policy artifacts colliding, guard duration, record re…
LaZzyMan Aug 12, 2026
d9eb5e3
test(omni): keep '+' in generated artifact stems
LaZzyMan Aug 12, 2026
4198602
feat(omni): re-anchor remembered media, and say when recall was trunc…
LaZzyMan Aug 12, 2026
423e81f
feat(omni): tell the model what a media handle is and to recall befor…
LaZzyMan Aug 12, 2026
8f593ad
fix(omni): close the honesty and hygiene gaps found reviewing S5 memory
LaZzyMan Aug 12, 2026
caf8244
fix(omni): disclose the session handle for URL-delivered media too
LaZzyMan Aug 12, 2026
53b2312
docs(omni): record the two recall fields that keep a page honest
LaZzyMan Aug 12, 2026
162beba
test(omni): pin the selector gate and the handle disclosure
LaZzyMan Aug 12, 2026
2d66b10
test(omni): pin the memory invariants that mutation probes walked thr…
LaZzyMan Aug 12, 2026
5a19093
fix(omni): keep a withheld resource reachable, and stop losing why re…
LaZzyMan Aug 12, 2026
78f46a8
test(omni): cover the four seams where memory reaches the pipeline
LaZzyMan Aug 12, 2026
e60efd8
fix(omni): keep S5 off the root barrel, which was dragging 550 KB int…
LaZzyMan Aug 12, 2026
f920897
fix(omni): anchor URL media off its staging file, and record ACP call…
LaZzyMan Aug 13, 2026
cf95fb3
docs(omni): reconcile storage lifecycle with the shipped recovery, st…
LaZzyMan Aug 13, 2026
ea4294b
test(omni): stop asserting an exact timeout the wall clock can undercut
LaZzyMan Aug 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/design/2026-07-29-omni-multimodal-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,23 @@ interface MediaMemoryRecallRequest {
召回返回的可用媒体衍生物会绑定出当前 session 的 `resourceId`,Agent 随后可把它传给
另一个 MediaPolicyTool。召回本身不复制文件、不创建新 File,也不触发 policy。

#### 9.2.1 "按路径/内容 hash 查询"的一致解读

issue #8188 同一条验收项里写了"按路径/内容 hash 查询"与"返回中永不暴露真实本地路径"。
按本设计 §5.2/§15 的路径隔离不变量,二者的一致解读是**按调用方分层**:

| 调用方 | 允许的入口 | 理由 |
| ------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 模型(active Tool / passive selector) | 仅 session `resourceId` | 句柄是不可伪造的能力凭证:模型只能问本会话真的投递给它的媒体。放开路径入参会让媒体文件里的注入内容驱使模型探测任意路径,而召回返回的是转写正文——典型 confused-deputy 提权;即使 miss,"有/无记忆"本身也是存在性泄露 |
| Harness(受管管线内部) | 路径 / 内容 hash | 不经模型,无提权面。已由 `findBindingBySha256` 提供并被反应式降质阶梯使用 |

因此"系统能按路径/hash 找到记忆"这一能力是满足的,只是不向模型暴露。

代价与已知缺口:句柄只在投递时签发,而投递要求文件存在——所以在新 session 中,被
删除或移动的文件无法再获得句柄,其记忆暂时不可达。补法不是放开模型的路径入参,而是
在**授权层**补:让 `@` 引用一个"记忆里有、磁盘上没了"的文件也能凭记录的
`fileRef`+`sha256` 签发句柄(用户的 `@` 引用即授权)。留作后续阶段。

### 9.3 被动 side-query 召回

被动召回只处理当前请求已经显式引用并完成识别的文件。它不从普通文本猜路径,也不
Expand Down Expand Up @@ -551,6 +568,12 @@ interface MediaMemoryRecallResult {
kind: 'metadata' | 'derived_media' | 'policy_result' | 'execution';
role?: string;
content?: string;
/** content 是 memory 所存文本的前缀时为 true(被 recall.maxTextChars 截断)。
* coverage 讲的是"处理过什么",所以一条被截断的转写仍合法地报 complete;
* 没有这个标记,模型看到前缀 + 完整覆盖 + 无 gap,就会对它从未读到的后半段
* 音频作答。默认值本身就会撞上:maxTextChars(24000 字符)小于收集期界
* (65536 字节),任何长转写都在读取期被切。 */
contentTruncated?: boolean;
resourceId?: string;
scope: MediaScope;
channels: MediaChannel[];
Expand Down Expand Up @@ -578,6 +601,10 @@ interface MediaMemoryRecallResult {
arguments: Record<string, unknown>;
reason: string;
}>;
/** 匹配总数,仅当条目预算把列表截短时出现。没有它,一页被截断的结果与穷尽结果
* 无法区分:真实审计在 limit: 12 下读到 6 段 clip,就得出"从未抽过关键帧",
* 而库里有 72 条——读的人没有说谎,它只是无从知道自己看到的是一页。 */
matchedEntries?: number;
}
```

Expand Down Expand Up @@ -705,6 +732,31 @@ managed auto-memory 配置,也不增加独立配置文件、环境变量或 CL
复用时每个 File 仍保留自己的 root、source 和 lineage 引用,不能把一个文件的
权限或 provenance 泄漏给另一个文件。

#### 11.2.1 三层身份:为什么"不合并"与"不重复建节点"并不矛盾

issue #8188 的验收项写作"内容相同的两个文件不重复建节点(复用同一底层身份)",
与本节标题字面冲突。二者实际落在**不同层**,实现同时满足:

| 层 | 键 | 承载 | 满足 |
| ----------- | ------------------ | --------------------------------------------------------------- | ---------------------------------------- |
| Content | `sha256` | 物理对象(`objects/sha256/…`)、可复用的计算结果与 policy 派生物 | #8188「不重复建节点(复用同一底层身份)」 |
| File | locator | `CURRENT_VERSION`、source、provenance、权限/workspace 归属 | 本节「不合并不同文件」、#8189 版本隔离 |
| FileVersion | `(fileId, sha256)` | 二者的连接,不可变 | 11.1 版本链 |

关键约束:**身份键不能是内容**。若 `fileId = hash(content)`,则"内容变化"直接
产生另一个 `fileId`,11.1 的版本链(新 FileVersion → 更新 `CURRENT_VERSION` →
旧版本留作历史 → 默认排除旧版本)与 #8189 验收项"修改文件内容后旧衍生物不默认
召回"**都无法表达**——编辑文件会凭空出现无关节点,改回去则历史成环。因此逻辑文件
必须由跨内容稳定的 locator 定键。

于是"不重复付费"由 content 层承担:同字节的第二个文件不重跑任何工具(见 11.3 的
复用键与 `reusedExecutionId`),不重复存储字节(共享同一 `objects/` 对象),只额外
写几行属于自己的廉价元数据行。

派生物同样按 `(rootFileId, objectPath)` 定键而非仅 `objectPath`:两个 root 派生出
逐字节相同的产物时,若共享一个 File 节点,该节点的 `rootFileId` 只能属于先创建者,
既泄漏 lineage 又让第二个 root 的有界遍历(§8)够不到它。

### 11.3 Policy 结果复用键

可复用 policy 结果的复用键必须与 11.2 的跨文件复用语义一致,因此以**内容身份**
Expand Down
55 changes: 33 additions & 22 deletions docs/design/2026-07-30-omni-managed-media-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,30 +140,41 @@

## 6. 生命周期与垃圾回收

### 6.1 启动恢复

每次启动(或首次触碰 Omni 链路时)执行一次恢复扫描:

1. 删除 `staging/` 下所有目录——它们属于未完成的 invocation,对应的
`OmniPolicySucceeded` 必然未提交(提交成功即已删除 staging),Memory 中不
存在引用;
2. 删除超过保留期的 `downloads/*.part`(续传窗口内的保留);
3. 按预算清理 `quarantine/`;
4. 校验 `objects/` 中随机抽样对象的文件名与实际 hash 一致(廉价的损坏探测,
不做全库校验)。

### 6.2 mark-and-sweep GC

- **根集合**:Memory store 中全部 active 记录引用的 managedId(含历史版本仍
被 provenance 引用的对象)+ 当前运行 session 的 MediaResourceRegistry 正在
使用的对象;
- **清扫对象**:不在根集合中、且自晋升起超过 `retentionDays` 的对象;宽限期
保证"晋升成功但 Memory 提交失败"的孤儿和跨进程 race 不被立刻误删;
### 6.1 启动恢复(已实现,`omni/recovery.ts`)

每次启动(或首次触碰 Omni 链路时)执行一次恢复扫描。实现相对初稿的修正——
多进程共存迫使"删除所有 staging"退化为宽限期语义:

1. 删除 `staging/` 下**超过宽限期(1h)**的目录——宽限期必须长于策略工具
最长超时(配置校验强制 `runtime.timeoutMs` 低于宽限期),否则第二个 CLI
进程的恢复扫描会把另一个进程正在转码的工作目录删掉。晋升半成品 `.tmp`
同理(1h 宽限);
2. 删除超过保留窗口(48h)的 `downloads/*.part`;无续传逻辑,窗口纯为事后
检查中断下载留的调试期;
3. 按保留天数与容量预算清理 `quarantine/`(超预算最旧优先);
4. 抽样校验 `objects/` 对象名与实际 hash 一致(每次至多 3 个、单个 ≤64MB,
避免在首次投递前的内联扫描里 hash 多 GB 视频);发现损坏对象删除时**级联
清理降质缓存**(`policy-cache.json` 中以它为源或产物的条目),防止缓存
命中一个永远无法投递的对象。

### 6.2 mark-and-sweep GC(S6 落点)

- **根集合**:Memory store 中全部记录引用的 managedId——`entries[].artifactRef.managedId`
与 `versions[].source.locator`(`protocol: 'managed'`,tool/URL 来源媒体的
身份锚,见 Memory 设计 §11.2.1 实现注记)两处都算,含仅被 provenance 引用的
历史版本对象——加上当前进程 MediaResourceRegistry 正在使用的对象;
- **清扫对象**:不在根集合中、且自晋升起超过 `retentionDays` 的对象;保留期
兼作宽限期,保证"晋升成功但 Memory 提交失败"的孤儿和跨进程 race 不被立刻
误删;
- **触发时机**:启动恢复后异步执行;超过 `maxTotalBytes` 时提前触发,仍超限
则从最旧的无引用对象继续删除;**有引用对象永不删除**,即使超预算——此时
告警并停止新的衍生物产生(等价于 Policy 设计 §8.4 的预算停止语义);
- GC 与 Memory 的一致性:先从 Memory 确认无引用,再删除文件;不存在"先删
文件再改记录"的窗口。
告警并置"停止新衍生物"标志,由 orchestrator 消费(等价于 Policy 设计 §8.4
的预算停止语义);
- **级联**:对象删除联动清理其 `upload-cache.json` 条目与降质缓存条目
(复用恢复扫描已实现的级联,见 §6.1 第 4 条);反向不成立;
- GC 与 Memory 的一致性:先从 Memory 快照确认无引用,再删除文件;不存在
"先删文件再改记录"的窗口。快照读取失败时 GC 整体跳过(fail-closed:
读不到根集合就不删任何东西)。

## 7. 配置

Expand Down
39 changes: 39 additions & 0 deletions packages/cli/src/acp-integration/session/Session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8191,6 +8191,45 @@ export class Session implements SessionContext {
? 'error'
: 'success';
const succeeded = status === 'success';
// ACP runs its own tool executor, so it must capture the policy
// artifact batch itself and call the SAME core memory boundary
// the scheduler and the fixed-policy orchestrator use (memory
// design M §17). Captured from the tool's OWN result, before any
// PostToolUse hook artifacts are merged below — hook artifacts
// must never impersonate policy outputs. Never throws; a
// collection failure cannot affect the tool result (D12).
if (
succeeded &&
tool.mediaPolicyDescriptor &&
toolResult.artifacts &&
toolResult.artifacts.length > 0
) {
// Deep subpath, not the root or `omni` barrel: both are
// statically imported across the CLI, so re-exporting this
// through either drags the whole policy graph — and, via
// iconvHelper's top-level iconv-lite import, its ~550 KB of
// encoding tables — into the ACP agent's static closure
// (scripts/check-serve-fast-path-bundle.js enforces that).
const { collectModelPolicyCall } = await import(
'@qwen-code/qwen-code-core/omniPolicyCollection'
);
await collectModelPolicyCall({
config: this.config,
batch: {
toolName,
invocationId: callId,
// Same pin as the modelAccess gate above: every
// ACP-originated call is a model call. Recording 'client'
// here made the PolicyExecution provenance contradict the
// gate that admitted the very same call.
executionOrigin: { kind: 'model' },
artifacts: toolResult.artifacts,
},
descriptor: tool.mediaPolicyDescriptor,
args,
signal: activeToolAbortSignal ?? abortSignal,
});
}
const responseError = toolResult.error
? new Error(toolResult.error.message)
: aborted
Expand Down
43 changes: 43 additions & 0 deletions packages/cli/src/config/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import { hooksCommand } from '../commands/hooks.js';
import { normalizeDisabledToolList } from './normalizeDisabledTools.js';
import type { LoadedSettings, Settings } from './settings.js';
import { loadSettings, SettingScope } from './settings.js';
import { getSettingsSchema } from './settingsSchema.js';
import {
resolveCliGenerationConfig,
getAuthTypeFromEnv,
Expand Down Expand Up @@ -1476,6 +1477,44 @@ export function buildDisabledSkillNamesProvider(
return () => resolveSkillSettings(loadedSettings).disabledNames;
}

/**
* Reject unknown keys directly under `omni`.
*
* The generic settings loader only scans TOP-LEVEL keys, and only writes a
* debug line — so a nested typo is caught by nothing: `omni.memoryy` or
* `omni.processingg` leaves `settings.omni?.memory` / `?.processing`
* undefined, every downstream normalizer sees "not configured" and returns
* defaults, and the session silently runs with the operator's entire
* configuration discarded (probe: `omni.memoryy.recall.mode = sideQuery`
* still registered the active-mode recall tool). The omni namespace's
* declared stance is that a misconfiguration must fail loud, so this
* mirrors the nested checks its own normalizers already perform.
*
* The allowed set is derived from the settings schema rather than
* hardcoded, so it cannot drift as the namespace grows.
*/
function assertKnownOmniSettingKeys(settings: Settings): void {
const omni = settings.omni;
if (omni === undefined || omni === null || typeof omni !== 'object') return;
const schemaOmni = getSettingsSchema()['omni'] as
| { properties?: Record<string, unknown> }
| undefined;
const allowed = new Set(Object.keys(schemaOmni?.properties ?? {}));
// No schema properties resolved (unexpected): stay silent rather than
// rejecting every valid key.
if (allowed.size === 0) return;
const unknown = Object.keys(omni).filter((key) => !allowed.has(key));
if (unknown.length > 0) {
throw new Error(
`Invalid settings: unknown key(s) under "omni": ` +
`${unknown.map((k) => `"${k}"`).join(', ')}. ` +
`Allowed: ${[...allowed].sort().join(', ')}. ` +
`An unrecognized omni section would be silently ignored, leaving ` +
`the session running with defaults instead of your configuration.`,
);
}
}

export async function loadCliConfig(
settings: Settings,
argv: CliArgs,
Expand Down Expand Up @@ -1521,6 +1560,7 @@ export async function loadCliConfig(
*/
settingsWatcher?: { stopWatching(): void },
): Promise<Config> {
assertKnownOmniSettingKeys(settings);
const debugMode = isDebugMode(argv);
if (debugMode && process.env['QWEN_DEBUG_LOG_FILE'] === undefined) {
process.env['QWEN_DEBUG_LOG_FILE'] = '1';
Expand Down Expand Up @@ -2197,6 +2237,8 @@ export async function loadCliConfig(
settings.omni?.processing?.transportGuard?.maxUploadFileBytes,
omniMaxEstimatedTokens:
settings.omni?.processing?.transportGuard?.maxEstimatedTokens,
omniMaxDurationSeconds:
settings.omni?.processing?.transportGuard?.maxDurationSeconds,
omniUrlDownloadMaxFileBytes:
settings.omni?.ingestion?.localization?.url?.maxFileBytes,
omniUploadUrlTtlHours: settings.omni?.delivery?.upload?.urlTtlHours,
Expand All @@ -2214,6 +2256,7 @@ export async function loadCliConfig(
omniQuarantineRetentionDays:
settings.omni?.storage?.quarantine?.retentionDays,
omniQuarantineMaxBytes: settings.omni?.storage?.quarantine?.maxBytes,
omniMemory: settings.omni?.memory as Record<string, unknown> | undefined,
// CDP tunnel (Plan C, #5626): with the tunnel on, browser automation goes
// through the CDP tunnel (far lighter than the OS-level computer-use
// driver), so disable computer-use to keep the agent off that heavy path.
Expand Down
Loading
Loading