diff --git a/docs/design/2026-07-09-webshell-user-message-input-annotations.md b/docs/design/2026-07-09-webshell-user-message-input-annotations.md new file mode 100644 index 00000000000..0fb41dbb738 --- /dev/null +++ b/docs/design/2026-07-09-webshell-user-message-input-annotations.md @@ -0,0 +1,396 @@ +# WebShell user message input annotations + +## 背景 + +WebShell 的 `@` 能力已经支持在输入框中把选中的文件、扩展、MCP 资源以及 host 自定义 provider 项渲染为 chip。输入框内的 chip 来自 CodeMirror inline widget,widget 持有完整的 `WebShellComposerTag`,因此可以稳定拿到 `id`、`kind`、`label`、`value`、`serialized`、`removable` 以及 host 通过 `composerTagIcons` 注入的 icon。 + +当前 PR1 的第一版实现没有改变发送链路,只在用户消息渲染阶段从 `content` 文本中重新解析 `@...` 引用,再把能够识别的 built-in 引用渲染为 chip。这解决了部分可逆场景,例如 `@.qwen/`、`@ext:name`、`@mcp:name`,但它依赖文本猜测,无法覆盖所有真实输入。 + +review 反馈暴露了这个方向的根本问题: + +- `@Makefile`、`@LICENSE`、`@src/Makefile` 是合法文件引用,但单靠文本无法和普通 mention 或 package-like token 稳定区分。 +- `@dataset:users` 这类 custom provider 引用在发送后只剩文本,默认渲染拿不到原始 `kind`、`label`、`value` 和 icon。 +- escaped MCP resource 与尾随标点的边界只能通过启发式处理,继续补规则会让 parser 越来越复杂,仍然无法证明完整正确。 + +因此 PR1 需要扩大范围:在不改变模型收到的 prompt 文本的前提下,把 composer 已经拥有的结构化输入 metadata 沿提交、transcript、本地消息和回放链路保存下来。用户消息渲染只使用 metadata 渲染 chip;旧消息或缺失 metadata 的消息保持原始文本显示,不再尝试从纯文本猜测引用。 + +这里不能把新增字段命名为 `composerTags`。`composerTag` 是当前 `@` chip 的实现细节,但 WebShell 的用户输入里还有 `/` slash command、skill command、custom command、system command、local command 等结构化输入。新的发送 metadata 应该表达“用户输入中的结构化注解”,本期只写入 `@` reference 注解,后续可以在同一字段中加入 `/` command 注解。 + +## 目标 + +- 用户在输入框中看到的 `@` reference chip,发送后在用户消息气泡中保持一致的 chip 渲染。 +- 支持 built-in file、extension、MCP tags,包括无扩展名文件和 escaped MCP resource。 +- 支持 host 自定义 provider 的默认 chip 渲染,只要 provider 在 accepted item 中提供了 `composerTag`。 +- 保持模型侧 prompt 内容不变,daemon/model 仍然接收当前 `buildComposerPrompt(text, tags)` 生成的字符串。 +- 保持 `renderUserMessageContent` 的覆盖能力;host 如果自定义了用户消息内容,仍然可以完全接管渲染。 +- 对旧 transcript、旧 daemon、无 metadata 消息保持兼容:内容仍原样显示,只是不额外渲染 chip。 +- 为 `/` command、skill command、custom command 等后续结构化输入预留统一扩展点。 + +## 非目标 + +- 不改变 `@` provider 注册协议。 +- 不新增 skill 的 `@skill:` 支持;WebShell 当前通过 `/` 引用 skill。 +- 不把 icon URL 写入持久化 transcript。icon 继续由 `composerTagIcons` 按 `kind` 在渲染时解析。 +- 不把 metadata 传给模型,也不改变 daemon prompt 解析语义。 +- 不尝试从纯文本 100% 还原所有 custom provider 或无扩展名文件引用。 +- 本期不改变 `/` command 的渲染;只把 metadata 字段设计成可以承载 `/` command 注解。 +- 本期不补 Ctrl+Y retry 的 annotation 重建;retry 复用原始用户消息,不新增重复 user echo。 +- 本期不补 `onSubmitBefore` 失败后的 annotation 回滚;失败时 prompt 不进入发送链路,保持当前取消行为。 + +## 范围决策 + +- 本期接受同时修改 `packages/web-shell`、`packages/webui`、`packages/sdk-typescript` 和 `packages/acp-bridge`。前三者负责提交、本地 echo、transcript/message 类型和渲染;`packages/acp-bridge` 负责把 daemon user echo 写入可 replay 的 `user_message_chunk.update._meta`,否则刷新/重开 session 后无法恢复 annotation。 +- 普通发送和 queued prompt 都需要支持 annotation。queued prompt 也会在用户消息区域显示本次输入,如果不携带 metadata,会和普通发送出现不一致。 +- `renderUserMessageContent` 需要扩展入参,让 host 自定义 renderer 可以读取 `inputAnnotations`。默认 renderer 使用 metadata 渲染 chip;host renderer 仍然拥有最终覆盖权。 +- 删除从纯文本推断 `@` chip 的 fallback,避免继续维护无法完整正确的启发式 parser。 +- 本期只生成和渲染 `@` reference annotation;`/` command、skill command、custom command 只在数据结构上预留,不实现发送后 chip 渲染。 + +## 已检索到的结构化输入能力 + +当前 WebShell 输入侧至少有以下结构化能力: + +- `@` references:由 `useAtMentionMenu` 提供,包含 built-in file、extension、MCP server/resource,以及 host 通过 `atProviders` 注入的自定义 provider。接受后会生成 `WebShellComposerTag`,并由 CodeMirror inline widget 渲染 chip。 +- `/` slash commands:由 `slashCompletion.ts` 提供补全。顶层 command 来自 daemon 的 `session.available_commands`、WebShell local commands、custom commands、skill commands 和 system commands。 +- `/` subcommands:`slashCompletion.ts` 支持显式 `subcommands`、内置 subcommand tree、implicit subcommand tree。例如 `/mcp desc`、`/stats model`、`/memory show`、`/skills `。 +- command category:`commandDisplay.ts` 把 command 分为 `custom`、`skill`、`system`。`App.tsx` 会根据 `connection.skills` 把对应 command 标记为 skill category。 +- local slash commands:`localCommands.ts` 中定义了 `help`、`theme`、`language`、`model`、`mcp`、`skills`、`memory`、`context`、`agents`、`goal`、`tasks`、`extensions` 等本地命令。 +- shell mode / `!`:composer 可以以 shell mode 提交 `!${prompt}`,这是另一种用户输入语义,但不在本期渲染范围。 + +这些能力说明新增 metadata 字段应该是通用 annotation 列表,而不是只服务 `@` 的 tag 列表。 + +## 现状链路 + +### 输入框内 + +`useComposerCore` 在输入框中维护 inline tags。提交时已经能通过 `tagsOverride ?? composerTagsRef.current` 拿到完整 `WebShellComposerTag[]`。这些 tag 用于 `buildComposerPrompt(text, tags)`,最终合并进发送给 daemon 的 prompt 文本。 + +### 发送和本地 echo + +`App.tsx` 的 `sendPrompt` 只接收 `text` 和 `images`,`sessionActions.sendPrompt(text, options)` 也只发送 prompt 文本。WebShell 为了乐观显示或本地命令 echo,会调用 `store.appendLocalUserMessage(text, images)`。 + +`appendLocalUserMessage` 目前只把 `text/images` 写入 `DaemonTextTranscriptBlock`,没有携带结构化输入 metadata。 + +### 回放到消息组件 + +`transcriptBlocksToDaemonMessages` 把 transcript user block 转成 `DaemonUserMessage`,当前只保留 `content`、`images`、`timestamp` 和 `source`。`UserMessage` 只能拿到 `content/images`,因此第一版实现只能通过文本 parser 重新猜测 tag。 + +## 方案概述 + +新增一条 UI-only metadata 链路。它分成两条相邻但职责不同的路径:当前页面的乐观 echo,以及 daemon transcript 的持久化 echo。 + +```text +CodeMirror inline tags + -> submitText / submitPromptFromEditor + -> sendPrompt options + -> sessionActions.sendPrompt / sessionActions.submitPrompt options + -> A. store.appendLocalUserMessage(text, images, { inputAnnotations }) + -> 当前 tab 立即显示用户消息 chip + -> B. PromptRequest._meta.inputAnnotations + -> bridge echoPromptToSessionBus 合并到 user_message_chunk.update._meta + -> replay/load 得到同一批 session_update 事件 + -> normalizeDaemonEvent 生成 user.text.delta.meta.inputAnnotations + -> reduceDaemonTranscriptEvents 写入 DaemonTextTranscriptBlock.meta.inputAnnotations + -> transcriptBlocksToDaemonMessages + -> DaemonUserMessage.inputAnnotations + -> UserMessage default renderer +``` + +`content` 仍然是模型和 daemon 需要处理的 prompt 文本。`inputAnnotations` 只描述 UI 渲染需要的结构化输入,不参与模型输入。 + +## 数据结构 + +新增通用输入注解结构,顶层字段命名为 `inputAnnotations`: + +```ts +interface DaemonUserMessage { + id: string; + role: 'user'; + content: string; + images?: Array<{ data: string; mimeType: string }>; + source?: string; + inputAnnotations?: DaemonInputAnnotation[]; +} +``` + +`DaemonInputAnnotation` 表达“content 中某一段文本对应的结构化语义”。设计原则是只新增外层 annotation wrapper,内部 payload 尽量复用现有 `@` 和 `/` 的对象格式,避免出现一套与 `WebShellComposerTag`、`CommandInfo` 平行的新协议。本期只落 `type: 'reference'`,后续 `/` command 可以复用同一个数组继续扩展: + +```ts +interface DaemonInputReferenceAnnotation { + type: 'reference'; + start: number; + end: number; + text: string; + reference: DaemonInputReference; +} + +interface DaemonInputReference { + id: string; + kind?: string; + label?: string; + value?: string; + serialized?: string; + removable?: boolean; +} + +type DaemonInputAnnotation = DaemonInputReferenceAnnotation; +``` + +`start/end` 是相对最终 `content` 的 UTF-16 offset,与 React/CodeMirror 当前字符串处理一致。它避免后续渲染再靠 `serialized` 从 `content` 中反查位置,也为多个相同引用、相同 command、inline 文本混排留下空间。 + +本期 `@` reference payload 直接复用现有 `WebShellComposerTag`: + +```ts +interface WebShellComposerTag { + id: string; + kind?: string; + label?: string; + value?: string; + serialized?: string; + removable?: boolean; +} +``` + +未来 `/` command payload 直接复用现有 `CommandInfo`,只在 annotation 层补 `subcommandPath`: + +```ts +interface CommandInfo { + name: string; + description: string; + argumentHint?: string; + subcommands?: string[]; + source?: string; + displayCategory?: 'custom' | 'skill' | 'system'; +} +``` + +在 SDK transcript block 的 `meta` 中存储同样的 `inputAnnotations`: + +```ts +interface DaemonTextDeltaMeta { + inputAnnotations?: DaemonInputAnnotation[]; +} +``` + +实现时 SDK 包不应该 import WebShell client 类型。SDK 中定义与 `WebShellComposerTag`、`CommandInfo` 字段兼容的最小 meta 结构,WebShell adapter 再把该结构转换成 client 渲染所需类型。这样可以避免 SDK 反向依赖 WebShell,同时保持字段形态与现有 `@` / `/` 格式一致。 + +## 关键修改点 + +### 1. 提交链路携带 inputAnnotations + +调整 editor submit 的参数形态,让 `sendPrompt` 能拿到提交时的 `DaemonInputAnnotation[]`。 + +建议新增轻量 options 字段: + +```ts +interface SendPromptInputMetadata { + inputAnnotations?: DaemonInputAnnotation[]; +} +``` + +`useComposerCore.submitText()` 在生成 prompt 文本时已经知道 `tags` 和最终 `prompt`。它需要把本期 `@` tags 转换成 `reference` annotations,再调用上层 `onSubmit`: + +- `promptText`: 当前发送给 daemon 的文本,保持不变。 +- `images`: 当前图片。 +- `inputAnnotations`: 提交瞬间的结构化输入注解快照。 + +如果当前 `onSubmit` 签名不适合直接扩展,可以新增第四个 metadata 参数,避免破坏已有调用: + +```ts +onSubmit(promptText, images, commitAccepted, { inputAnnotations }); +``` + +本期 annotation 生成规则: + +- 对 `buildComposerPrompt(text, tags)` 生成的 tag prefix 计算 `start/end`。 +- 每个 tag 对应一个 `type: 'reference'` annotation。 +- `annotation.text` 使用最终 prompt 中的实际 serialized 文本。 +- `annotation.reference` 保存原有 `WebShellComposerTag` 的最小安全字段:`id/kind/label/value/serialized/removable`。 +- 不保存 icon URL;icon 仍由 `kind + composerTagIcons` 在渲染时解析。 + +如果将来 `/` command 也需要结构化渲染,可在 slash completion accept 时生成 `type: 'command'` annotation,或者在 submit 阶段根据命中的 `CommandInfo` 生成 command annotation。command payload 直接保存现有 `CommandInfo` 字段,subcommand 信息放在 annotation wrapper 的 `subcommandPath` 中。 + +### 2. 本地 transcript echo 保存 metadata + +扩展 SDK transcript store: + +```ts +appendLocalUserMessage( + text: string, + images?: Array<{ data: string; mimeType: string }>, + meta?: { inputAnnotations?: DaemonInputAnnotation[] }, +): void; +``` + +`appendLocalUserTranscriptMessage` 同步接收 `meta`: + +```ts +appendLocalUserTranscriptMessage(state, text, { images, meta }); +``` + +创建 user text block 后写入: + +```ts +if (opts.meta) { + block.meta = { ...block.meta, ...opts.meta }; +} +``` + +这条链路只保证当前前端 store 中的乐观用户消息立即带 chip。它不单独保证刷新或重新打开 session 后仍能拿到 metadata,因为刷新后的 transcript 来自 daemon replay,而不是当前 tab 内存中的 local append。 + +没有 input annotations 的本地 slash command 继续传空 metadata,不改变现有行为。 + +### 3. daemon prompt echo 持久化 metadata + +`PromptRequest` 当前已经支持 `_meta?: Record | null`。发送时把同一份 `inputAnnotations` 写入 `PromptRequest._meta.inputAnnotations`: + +```ts +const promptRequest = { + prompt: toDaemonPromptContent(text, normalizedImages), + _meta: inputAnnotations.length > 0 ? { inputAnnotations } : undefined, +}; +``` + +bridge 在 `sendPrompt` 内会把 request 交给 agent prompt,同时通过 `echoPromptToSessionBus` 发布 `user_message_chunk`。这里需要把 request `_meta.inputAnnotations` 合并到 echo 的 `update._meta` 中: + +```ts +_meta: { + ...pickUserInputEchoMeta(req._meta), + serverTimestamp, + source: 'bridge-echo', +} +``` + +`pickUserInputEchoMeta` 只保留 `inputAnnotations`,不把未知 request meta 原样写入用户消息 transcript。这样可以避免把 telemetry、requestId、retry 等非 UI 数据暴露给 `UserMessage`。 + +replay 时,`DaemonSessionProvider` 会把 `compactedReplay/liveJournal` 重新 normalize 成 UI events;`normalizeDaemonEvent` 已经会把 `user_message_chunk.update._meta` 放到 `user.text.delta.meta`;transcript reducer 已经会把 text event 的 `meta` 写入 `DaemonTextTranscriptBlock.meta`。因此只要 daemon echo 事件里带上 `inputAnnotations`,刷新和重新打开同一 session 后就能恢复 chip 渲染。 + +### 4. transcript adapter 转发 metadata + +`transcriptBlocksToDaemonMessages` 当前已经读取 user block 的 `meta.source`。在同一位置读取 `meta.inputAnnotations`,校验为数组后写到 `DaemonUserMessage.inputAnnotations`。 + +这里需要做最小结构校验,避免 transcript 中的未知 meta 影响渲染: + +- 必须是数组。 +- 每个 annotation 必须有非空 string `id/type/text`。 +- `start` 和 `end` 必须是有限数字,且满足 `0 <= start < end <= content.length`。 +- 本期只生成并渲染 `type: 'reference'` 的 annotation;后续 command annotation 可以在同一字段下扩展。 +- reference payload 按 `WebShellComposerTag` 的字段做最小净化,只接受 `id/kind/label/value/serialized` 的 string 值和 `removable` 的 boolean 值。 +- command payload 按 `CommandInfo` 的字段做最小净化,只接受 `name/description/argumentHint/source/displayCategory` 的 string 值和 `subcommands` 的 string 数组。 +- 不保留未知字段。 + +### 5. UserMessage 优先使用 inputAnnotations + +`UserMessage` props 增加: + +```ts +inputAnnotations?: DaemonInputAnnotation[]; +``` + +`renderUserMessageContent` 入参同步增加同名字段: + +```ts +renderUserMessageContent?.({ content, images, inputAnnotations }); +``` + +默认渲染逻辑改成: + +1. 如果 `inputAnnotations` 中存在合法的 `type: 'reference'` 注解,按 `start/end` 切分 `content` 并渲染 chip。 +2. 如果 metadata 缺失或没有合法 annotation,直接渲染原始文本。 +3. 如果 host 提供 `renderUserMessageContent`,继续优先使用 host renderer。 + +metadata 渲染不再从 `content` 中猜 tag 类型,也不需要按 serialized 文本查找位置。range 非法或互相重叠时忽略对应 annotation,保证不隐藏任何用户内容。 + +### 6. 删除文本 parser fallback + +`splitComposerTagContent` 不再保留。原因是旧 parser 只能靠字符串形态猜测引用类型: + +- `@Makefile` 和 `@alice` 都可能是合法文本。 +- `@dataset:users` 需要 provider metadata 才知道 label/value/icon。 +- escaped MCP resource 的尾部标点很难靠通用规则证明正确。 + +因此默认用户消息只在 annotation 存在时渲染 chip;缺失 annotation 时展示原始文本。这样 review 中的 `@Makefile` 问题不再依赖启发式,因为新消息会从 metadata 获得明确的 file tag。 + +## Custom provider 行为 + +provider 如果在 accepted item 中提供: + +```ts +composerTag: { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', +} +``` + +发送后默认用户消息可以渲染: + +- label: `Dataset` +- value: `users` +- icon: 通过 `composerTagIcons.dataset` 解析 + +provider 如果没有提供 `composerTag`,发送后仍然只有纯文本,默认 renderer 不承诺自动识别 custom provider。host 仍然可以用 `renderUserMessageContent` 自行处理。 + +## 兼容性 + +- 旧 transcript 没有 `meta.inputAnnotations`,继续按原始文本显示。 +- 新 client 读取旧 daemon 事件时没有行为变化。 +- 旧 client 读取带 `meta.inputAnnotations` 的 transcript 时会忽略未知 meta。 +- `content` 不变,因此 daemon prompt 解析、模型输入、slash command 文本、历史 prompt 内容不受影响。 +- `renderUserMessageContent` 的优先级不变,host 自定义渲染不会被默认 chip 覆盖。 + +## 测试计划 + +### Unit tests + +- `appendLocalUserTranscriptMessage` 保存 `meta.inputAnnotations`。 +- `createDaemonTranscriptStore().appendLocalUserMessage` 能接收并保留 metadata。 +- `sessionActions.sendPrompt` 和 `sessionActions.submitPrompt` 能把 `inputAnnotations` 写入 `PromptRequest._meta`。 +- bridge `echoPromptToSessionBus` 只把 `inputAnnotations` 合并到 `user_message_chunk.update._meta`,不把未知 request meta 写入 transcript echo。 +- replay 的 `user_message_chunk.update._meta.inputAnnotations` 能经 `normalizeDaemonEvent` 和 reducer 写入 `DaemonTextTranscriptBlock.meta.inputAnnotations`。 +- `transcriptBlocksToDaemonMessages` 将 user block 的 `meta.inputAnnotations` 转成 `DaemonUserMessage.inputAnnotations`。 +- `transcriptBlocksToDaemonMessages` 过滤非法 annotation meta。 +- `UserMessage` 使用 reference annotation 渲染 `@Makefile`、`@LICENSE`、`@src/Makefile`。 +- `UserMessage` 使用 reference annotation 渲染 custom provider tag,并解析 `composerTagIcons`。 +- `UserMessage` 在 metadata 缺失时保持原始文本显示。 +- `UserMessage` 在 annotation range 非法或重叠时忽略该 annotation,不丢失原文。 +- 预留的 command annotation 类型可以被 schema 校验保留,但本期默认渲染忽略它,不影响 reference 渲染。 + +### Integration / browser verification + +- 在本地 WebShell 选择 `.qwen/`、`Makefile` 或 `LICENSE`,发送后用户消息仍显示 file chip。 +- 选择 MCP resource,发送后用户消息显示 MCP chip,resource 中的转义字符不被错误 trim。 +- 注入一个 custom provider,选择后发送,用户消息显示 custom label/value/icon。 +- 刷新页面或重新打开同一 session,用户消息 chip 仍然存在。 + +## 风险和控制 + +- 风险:跨包类型增加会扩大 PR 面积。控制方式是在 SDK 中定义最小 `DaemonInputAnnotation`,避免 SDK import WebShell client 类型。 +- 风险:metadata 与 `content` 不一致会导致渲染错位。控制方式是 UserMessage 只使用合法且不重叠的 range,非法 annotation 直接忽略,不隐藏任何用户内容。 +- 风险:持久化 custom provider 信息可能包含 host 自定义字段。控制方式是只保存 `id/kind/label/value/serialized/removable`,不保存未知字段和 icon URL。 +- 风险:PR1 范围扩大后 review 成本上升。控制方式是提交说明明确 motivation:这是为了解决纯文本 parser 无法正确还原 file/custom/MCP identity 的根因,同时保持 model-facing prompt 不变。 +- 风险:顶层 metadata 命名过窄会限制后续 `/` 能力。控制方式是使用 `inputAnnotations` 作为统一入口,本期只写入 `type: 'reference'`。 + +## 实施顺序 + +1. 在 SDK transcript 类型中增加 input annotation meta 的最小结构。 +2. 扩展 `appendLocalUserTranscriptMessage` 和 `DaemonTranscriptStore.appendLocalUserMessage`。 +3. 扩展 WebShell submit options,从 `useComposerCore` 到 `App.sendPrompt`、queued prompt submit 传递 `inputAnnotations`。 +4. 在乐观 echo 写入 `store.appendLocalUserMessage` 时带上 `inputAnnotations`。 +5. 在 daemon `PromptRequest._meta` 中写入 `inputAnnotations`,并让 bridge user echo 把它合并到 `user_message_chunk.update._meta`。 +6. 在 `transcriptBlocksToDaemonMessages` 中转发并净化 `meta.inputAnnotations`。 +7. 扩展 `DaemonUserMessage`、`MessageList` 到 `UserMessage` 的 props 链路。 +8. 扩展 `renderUserMessageContent` 入参,向 host renderer 暴露 `inputAnnotations`。 +9. `UserMessage` 默认渲染只使用 metadata;无 metadata 时原样文本显示。 +10. 补齐 unit tests 和浏览器验收截图。 + +## PR 描述要点 + +PR 描述需要说明: + +- 这不是改变模型 prompt,而是保存 WebShell 已经拥有的 UI input annotation metadata。 +- 纯文本 parser 无法可靠区分 `@Makefile`、`@alice`、`@dataset:users` 等形态,因此 metadata 是必要的。 +- 旧消息仍兼容为原始文本显示,custom provider 只有在提供 `composerTag` 时才享受默认 chip 渲染。 +- 新字段命名为 `inputAnnotations`,本期只承载 `@` reference,后续可以承载 `/` command、skill command、custom command 等结构化输入。 +- `renderUserMessageContent` 仍然是 host 的最终覆盖出口。 diff --git a/packages/acp-bridge/src/bridge.test.ts b/packages/acp-bridge/src/bridge.test.ts index 31007ced5bb..d6c7bc5d35d 100644 --- a/packages/acp-bridge/src/bridge.test.ts +++ b/packages/acp-bridge/src/bridge.test.ts @@ -4409,6 +4409,20 @@ describe('createAcpSessionBridge', () => { const aPromise = firstUserChunk(iterA); const bPromise = firstUserChunk(iterB); + const inputAnnotations = [ + { + type: 'reference', + start: 0, + end: 7, + text: '@file/a', + reference: { + id: 'file:@file/a', + kind: 'file', + value: 'file/a', + serialized: '@file/a', + }, + }, + ]; // Client A sends the prompt with its trusted clientId. await bridge.sendPrompt( @@ -4416,6 +4430,7 @@ describe('createAcpSessionBridge', () => { { sessionId: session.sessionId, prompt: [{ type: 'text', text: 'hello from A' }], + _meta: { inputAnnotations, privateRequestId: 'hidden' }, }, undefined, { clientId: session.clientId }, @@ -4443,6 +4458,12 @@ describe('createAcpSessionBridge', () => { expect((update._meta as { source?: string })?.source).toBe( 'bridge-echo', ); + expect( + (update._meta as { inputAnnotations?: unknown })?.inputAnnotations, + ).toEqual(inputAnnotations); + expect( + (update._meta as { privateRequestId?: unknown })?.privateRequestId, + ).toBeUndefined(); } abortA.abort(); diff --git a/packages/acp-bridge/src/bridge.ts b/packages/acp-bridge/src/bridge.ts index 72aa0e0712f..439159ae47a 100644 --- a/packages/acp-bridge/src/bridge.ts +++ b/packages/acp-bridge/src/bridge.ts @@ -767,6 +767,14 @@ function parseWorkspaceMemoryDreamResult( }; } +function pickUserInputEchoMeta(meta: unknown): Record { + if (!meta || typeof meta !== 'object' || Array.isArray(meta)) return {}; + const inputAnnotations = (meta as Record)[ + 'inputAnnotations' + ]; + return Array.isArray(inputAnnotations) ? { inputAnnotations } : {}; +} + /** * Echo a user prompt to the session bus so multi-client SSE subscribers * see the input alongside the agent response. Iterates content blocks @@ -830,7 +838,11 @@ function echoPromptToSessionBus( // field permitted alongside spec fields, the SDK normalizer // reads it from `update._meta`/`data._meta`, and every other // agent-emitted session_update carries `_meta` the same way. - _meta: { serverTimestamp, source: 'bridge-echo' }, + _meta: { + ...pickUserInputEchoMeta(req._meta), + serverTimestamp, + source: 'bridge-echo', + }, }, }, ...(originatorClientId ? { originatorClientId } : {}), diff --git a/packages/sdk-typescript/src/daemon/index.ts b/packages/sdk-typescript/src/daemon/index.ts index 738d27aec66..a49bbf3958e 100644 --- a/packages/sdk-typescript/src/daemon/index.ts +++ b/packages/sdk-typescript/src/daemon/index.ts @@ -131,6 +131,9 @@ export type { DaemonUserShellTranscriptBlock, DaemonPermissionTranscriptBlock, DaemonStatusTranscriptBlock, + DaemonInputAnnotation, + DaemonInputReference, + DaemonInputReferenceAnnotation, DaemonTextTranscriptBlock, DaemonTextDeltaMeta, DaemonToolPreview, diff --git a/packages/sdk-typescript/src/daemon/ui/index.ts b/packages/sdk-typescript/src/daemon/ui/index.ts index e5f9b1a2d7c..639b24b553f 100644 --- a/packages/sdk-typescript/src/daemon/ui/index.ts +++ b/packages/sdk-typescript/src/daemon/ui/index.ts @@ -66,6 +66,9 @@ export type { DaemonUserShellTranscriptBlock, DaemonPermissionTranscriptBlock, DaemonStatusTranscriptBlock, + DaemonInputAnnotation, + DaemonInputReference, + DaemonInputReferenceAnnotation, DaemonTextTranscriptBlock, DaemonTextDeltaMeta, DaemonToolPreview, diff --git a/packages/sdk-typescript/src/daemon/ui/store.ts b/packages/sdk-typescript/src/daemon/ui/store.ts index b9326593360..0aba9aa97f3 100644 --- a/packages/sdk-typescript/src/daemon/ui/store.ts +++ b/packages/sdk-typescript/src/daemon/ui/store.ts @@ -5,6 +5,7 @@ */ import type { + DaemonTextDeltaMeta, DaemonTranscriptState, DaemonTranscriptStore, DaemonUiEvent, @@ -60,8 +61,9 @@ export function createDaemonTranscriptStore( appendLocalUserMessage( text: string, images?: Array<{ data: string; mimeType: string }>, + meta?: DaemonTextDeltaMeta, ) { - state = appendLocalUserTranscriptMessage(state, text, { images }); + state = appendLocalUserTranscriptMessage(state, text, { images, meta }); scheduleNotify(); }, reset(nextSeed: Partial = {}) { diff --git a/packages/sdk-typescript/src/daemon/ui/transcript.ts b/packages/sdk-typescript/src/daemon/ui/transcript.ts index bdc8ac1a639..57d47c773e5 100644 --- a/packages/sdk-typescript/src/daemon/ui/transcript.ts +++ b/packages/sdk-typescript/src/daemon/ui/transcript.ts @@ -8,6 +8,7 @@ import type { DaemonPromptCancelledTranscriptBlock, DaemonShellTranscriptBlock, DaemonStatusTranscriptBlock, + DaemonTextDeltaMeta, DaemonTextTranscriptBlock, DaemonToolTranscriptBlock, DaemonTranscriptBlock, @@ -96,11 +97,19 @@ export function appendLocalUserTranscriptMessage( text: string, opts: DaemonTranscriptReducerOptions & { images?: Array<{ data: string; mimeType: string }>; + meta?: DaemonTextDeltaMeta; } = {}, ): DaemonTranscriptState { const next = cloneTranscriptState(state, opts); finishAssistant(next); - const block = createTextBlock(next, 'user', text); + const block = createTextBlock( + next, + 'user', + text, + undefined, + undefined, + opts.meta, + ); if (opts.images && opts.images.length > 0) { (block as DaemonTextTranscriptBlock).images = [...opts.images]; } diff --git a/packages/sdk-typescript/src/daemon/ui/types.ts b/packages/sdk-typescript/src/daemon/ui/types.ts index 0b7be648165..34a11c780eb 100644 --- a/packages/sdk-typescript/src/daemon/ui/types.ts +++ b/packages/sdk-typescript/src/daemon/ui/types.ts @@ -94,8 +94,28 @@ export interface DaemonUiTextEvent extends DaemonUiEventBase { meta?: DaemonTextDeltaMeta; } +export interface DaemonInputReference { + id: string; + kind?: string; + label?: string; + value?: string; + serialized?: string; + removable?: boolean; +} + +export interface DaemonInputReferenceAnnotation { + type: 'reference'; + start: number; + end: number; + text: string; + reference: DaemonInputReference; +} + +export type DaemonInputAnnotation = DaemonInputReferenceAnnotation; + export interface DaemonTextDeltaMeta extends Record { qwenDiscreteMessage?: boolean; + inputAnnotations?: DaemonInputAnnotation[]; } export interface DaemonUiUserImageEvent extends DaemonUiEventBase { @@ -963,6 +983,7 @@ export interface DaemonTranscriptStore { appendLocalUserMessage( text: string, images?: Array<{ data: string; mimeType: string }>, + meta?: DaemonTextDeltaMeta, ): void; reset(seed?: Partial): void; /** diff --git a/packages/sdk-typescript/test/unit/daemonUi.test.ts b/packages/sdk-typescript/test/unit/daemonUi.test.ts index 868c3485b4b..c02c0827e83 100644 --- a/packages/sdk-typescript/test/unit/daemonUi.test.ts +++ b/packages/sdk-typescript/test/unit/daemonUi.test.ts @@ -64,6 +64,72 @@ describe('daemon UI normalizer and transcript reducer', () => { ]); }); + it('stores input annotations on locally appended user messages', () => { + const store = createDaemonTranscriptStore(); + const inputAnnotations = [ + { + type: 'reference' as const, + start: 6, + end: 13, + text: '@file/a', + reference: { + id: 'file:@file/a', + kind: 'file', + value: 'file/a', + serialized: '@file/a', + }, + }, + ]; + + store.appendLocalUserMessage('hello @file/a', undefined, { + inputAnnotations, + }); + + expect(store.getSnapshot().blocks[0]).toMatchObject({ + kind: 'user', + meta: { inputAnnotations }, + }); + }); + + it('stores input annotations from replayed user message chunks', () => { + const inputAnnotations = [ + { + type: 'reference' as const, + start: 0, + end: 7, + text: '@file/a', + reference: { + id: 'file:@file/a', + kind: 'file', + value: 'file/a', + serialized: '@file/a', + }, + }, + ]; + const state = reduceDaemonTranscriptEvents( + createDaemonTranscriptState({ now: 1 }), + normalizeDaemonEvent({ + id: 1, + v: 1, + type: 'session_update', + data: { + sessionId: 's1', + update: { + sessionUpdate: 'user_message_chunk', + content: { type: 'text', text: '@file/a' }, + _meta: { inputAnnotations }, + }, + }, + }), + { now: 2 }, + ); + + expect(state.blocks[0]).toMatchObject({ + kind: 'user', + meta: { inputAnnotations }, + }); + }); + it('preserves assistant message metadata on transcript blocks', () => { const events = normalizeDaemonEvent({ id: 10, diff --git a/packages/web-shell/client/App.test.tsx b/packages/web-shell/client/App.test.tsx index 2b36f5201e7..569e802ff72 100644 --- a/packages/web-shell/client/App.test.tsx +++ b/packages/web-shell/client/App.test.tsx @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { act, createRef } from 'react'; import { createRoot, type Root } from 'react-dom/client'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; import type { WebShellApi } from './App'; type StreamingState = 'idle' | 'responding'; @@ -29,6 +30,7 @@ type ChatEditorTestProps = { text: string, images?: undefined, commitAccepted?: () => void, + metadata?: { inputAnnotations?: DaemonInputAnnotation[] }, ) => boolean | void; skills?: Array<{ name: string; description: string }>; isPreparing?: boolean; @@ -105,6 +107,7 @@ const { }, testState: { prompt: 'hello', + inputAnnotations: undefined as DaemonInputAnnotation[] | undefined, streamingState: 'idle' as StreamingState, blocks: [] as unknown[], latestChatEditorProps: null as ChatEditorTestProps | null, @@ -204,8 +207,15 @@ vi.mock('./components/ChatEditor', async () => { { 'data-testid': 'submit', 'data-preparing': props.isPreparing ? 'true' : 'false', - onClick: () => - props.onSubmit(testState.prompt, undefined, editorCommit), + onClick: () => { + if (testState.inputAnnotations) { + props.onSubmit(testState.prompt, undefined, editorCommit, { + inputAnnotations: testState.inputAnnotations, + }); + return; + } + props.onSubmit(testState.prompt, undefined, editorCommit); + }, type: 'button', }, 'submit', @@ -536,6 +546,7 @@ beforeEach(() => { mockConnection.loadingTranscript = false; mockConnection.catchingUp = false; testState.prompt = 'hello'; + testState.inputAnnotations = undefined; testState.streamingState = 'idle'; testState.blocks = []; testState.latestChatEditorProps = null; @@ -919,6 +930,7 @@ describe('App session callbacks', () => { 'queued', undefined, undefined, + undefined, ); expect(onSessionChange).toHaveBeenCalledWith({ type: 'submit', @@ -965,6 +977,35 @@ describe('App session callbacks', () => { expect(editorClear).not.toHaveBeenCalled(); }); + it('forwards input annotations for /plan prompts in active sessions', async () => { + const annotation: DaemonInputAnnotation = { + type: 'reference', + text: '@.husky/', + start: 0, + end: 8, + reference: { + id: '.husky/', + value: '.husky/', + serialized: '@.husky/', + }, + }; + const { container } = renderApp(); + await flush(); + + testState.prompt = '/plan @.husky/ explain'; + testState.inputAnnotations = [annotation]; + await clickSubmit(container); + await flush(); + + expect(mockSessionActions.setApprovalMode).toHaveBeenCalledWith('plan'); + expect(mockSessionActions.sendPrompt).toHaveBeenCalledWith( + '@.husky/ explain', + expect.objectContaining({ + inputAnnotations: [annotation], + }), + ); + }); + it('dispatches turn_complete only for the session that was streaming', async () => { const onSessionChange = vi.fn(); const { container, rerender } = renderApp({ onSessionChange }); diff --git a/packages/web-shell/client/App.tsx b/packages/web-shell/client/App.tsx index 968b2ebfd83..4bdac7cb022 100644 --- a/packages/web-shell/client/App.tsx +++ b/packages/web-shell/client/App.tsx @@ -26,6 +26,7 @@ import { } from '@qwen-code/webui/daemon-react-sdk'; import { isDaemonTurnError } from '@qwen-code/sdk/daemon'; import type { + DaemonInputAnnotation, DaemonTranscriptBlock, DaemonSessionTaskStatus, } from '@qwen-code/sdk/daemon'; @@ -298,6 +299,7 @@ interface ActiveGoalStatus { interface SendPromptOptionsWithRetry { optimisticUserMessage?: boolean; images?: PromptImage[]; + inputAnnotations?: DaemonInputAnnotation[]; retry?: boolean; clearComposerOnPromptStart?: boolean; commitComposerAccepted?: ComposerSubmitCommit; @@ -979,12 +981,12 @@ export function App({ }, []); const customization = useMemo( () => ({ + composerTagIcons, renderToolHeaderExtra, renderWelcomeHeader, renderWelcomeFooter, parseUserMessageContent, renderUserMessageContent, - composerTagIcons, renderComposerTag, renderComposerTagTooltip, onComposerTagClick, @@ -1000,12 +1002,12 @@ export function App({ loadingPhrases, }), [ + composerTagIcons, renderToolHeaderExtra, renderWelcomeHeader, renderWelcomeFooter, parseUserMessageContent, renderUserMessageContent, - composerTagIcons, renderComposerTag, renderComposerTagTooltip, onComposerTagClick, @@ -1791,6 +1793,7 @@ export function App({ opts?: { optimisticUserMessage?: boolean; retry?: boolean; + inputAnnotations?: DaemonInputAnnotation[]; clearComposerOnPromptStart?: boolean; commitComposerAccepted?: ComposerSubmitCommit; onAdmitted?: () => void; @@ -1848,6 +1851,7 @@ export function App({ } const promptOptions: SendPromptOptionsWithRetry = { images, + inputAnnotations: opts?.inputAnnotations, optimisticUserMessage: opts?.optimisticUserMessage, retry: opts?.retry, ...(opts?.onAdmitted ? { onAdmitted: opts.onAdmitted } : {}), @@ -1968,6 +1972,7 @@ export function App({ images?: PromptImage[], onComplete?: () => void, commitComposerAccepted?: ComposerSubmitCommit, + inputAnnotations?: DaemonInputAnnotation[], ) => { if (onSubmitBeforeRef.current) { onSubmitBeforeRef @@ -1976,7 +1981,12 @@ export function App({ prompt: text, }) .then(() => { - const result = rawEnqueuePrompt(text, images, onComplete); + const result = rawEnqueuePrompt( + text, + images, + onComplete, + inputAnnotations, + ); if (result !== false) { if (commitComposerAccepted) { commitComposerAccepted(); @@ -2002,7 +2012,12 @@ export function App({ }); return false; } - const result = rawEnqueuePrompt(text, images, onComplete); + const result = rawEnqueuePrompt( + text, + images, + onComplete, + inputAnnotations, + ); const sessionId = connectionRef.current.sessionId; if (sessionId && text.trim()) { dispatchSessionChangeRef.current?.({ @@ -3088,6 +3103,7 @@ export function App({ text: string, images?: PromptImage[], commitComposerAccepted?: ComposerSubmitCommit, + metadata?: { inputAnnotations?: DaemonInputAnnotation[] }, ) => { if (connectionRef.current.loadingTranscript) { pushToast('warning', t('editor.sessionLoading')); @@ -3106,7 +3122,11 @@ export function App({ promptText: string, promptImages: PromptImage[] | undefined, errorMessage: string, - opts?: { optimisticUserMessage?: boolean; retry?: boolean }, + opts?: { + optimisticUserMessage?: boolean; + retry?: boolean; + inputAnnotations?: DaemonInputAnnotation[]; + }, ) => { const deferComposerCommit = Boolean(onSubmitBeforeRef.current); const clearComposerOnPromptStart = @@ -3131,12 +3151,14 @@ export function App({ images, undefined, commitComposerAccepted, + metadata?.inputAnnotations, ); } return submitPromptFromEditor( text, images, 'Failed to send hidden slash command', + { inputAnnotations: metadata?.inputAnnotations }, ); } if (cmd === 'help') { @@ -3316,12 +3338,14 @@ export function App({ images, undefined, commitComposerAccepted, + metadata?.inputAnnotations, ); } return submitPromptFromEditor( text, images, 'Failed to send /model --fast', + { inputAnnotations: metadata?.inputAnnotations }, ); } if (modelArg === '--voice') { @@ -3391,6 +3415,7 @@ export function App({ prompt, images, 'Failed to send plan prompt', + { inputAnnotations: metadata?.inputAnnotations }, ); } return true; @@ -3403,6 +3428,7 @@ export function App({ if (prompt) { return sendPrompt(prompt, images, { clearComposerOnPromptStart: true, + inputAnnotations: metadata?.inputAnnotations, }).catch((error: unknown) => reportError(error, 'Failed to send plan prompt'), ); @@ -3491,12 +3517,14 @@ export function App({ images, undefined, commitComposerAccepted, + metadata?.inputAnnotations, ); } return submitPromptFromEditor( text, images, 'Failed to send /skills command', + { inputAnnotations: metadata?.inputAnnotations }, ); } else { if (echoOrDeferLocalCommand(text, images)) return true; @@ -3743,12 +3771,14 @@ export function App({ images, undefined, commitComposerAccepted, + metadata?.inputAnnotations, ); } return submitPromptFromEditor( text, images, 'Failed to send /rename command', + { inputAnnotations: metadata?.inputAnnotations }, ); } const displayName = renameArg.displayName; @@ -3938,9 +3968,17 @@ export function App({ } // Forward slash commands as prompts if (promptBlocked) { - return enqueuePrompt(text, images, undefined, commitComposerAccepted); + return enqueuePrompt( + text, + images, + undefined, + commitComposerAccepted, + metadata?.inputAnnotations, + ); } - return submitPromptFromEditor(text, images, 'Failed to send command'); + return submitPromptFromEditor(text, images, 'Failed to send command', { + inputAnnotations: metadata?.inputAnnotations, + }); } else if (text.startsWith('!')) { if (promptBlocked) { pushToast('error', t('queue.shellBlocked')); @@ -3955,9 +3993,17 @@ export function App({ return true; } else { if (promptBlocked) { - return enqueuePrompt(text, images, undefined, commitComposerAccepted); + return enqueuePrompt( + text, + images, + undefined, + commitComposerAccepted, + metadata?.inputAnnotations, + ); } - return submitPromptFromEditor(text, images, 'Failed to send message'); + return submitPromptFromEditor(text, images, 'Failed to send message', { + inputAnnotations: metadata?.inputAnnotations, + }); } }, [ @@ -4001,8 +4047,14 @@ export function App({ text: string, images?: PromptImage[], commitComposerAccepted?: ComposerSubmitCommit, + metadata?: { inputAnnotations?: DaemonInputAnnotation[] }, ) => { - const accepted = handleSubmit(text, images, commitComposerAccepted); + const accepted = handleSubmit( + text, + images, + commitComposerAccepted, + metadata, + ); if (accepted !== false) { resumeChatBottomFollow('smooth'); } diff --git a/packages/web-shell/client/adapters/messageTypes.ts b/packages/web-shell/client/adapters/messageTypes.ts index 4e7482802fe..e2a18de3129 100644 --- a/packages/web-shell/client/adapters/messageTypes.ts +++ b/packages/web-shell/client/adapters/messageTypes.ts @@ -4,6 +4,8 @@ * SPDX-License-Identifier: Apache-2.0 */ +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; + export type DaemonMessageToolCallStatus = | 'pending' | 'in_progress' @@ -79,6 +81,7 @@ export interface DaemonUserMessage extends DaemonMessageMeta { role: 'user'; content: string; images?: Array<{ data: string; mimeType: string }>; + inputAnnotations?: DaemonInputAnnotation[]; source?: string; } diff --git a/packages/web-shell/client/adapters/transcriptToMessages.test.ts b/packages/web-shell/client/adapters/transcriptToMessages.test.ts index cf8cdcaf105..751204cfbc4 100644 --- a/packages/web-shell/client/adapters/transcriptToMessages.test.ts +++ b/packages/web-shell/client/adapters/transcriptToMessages.test.ts @@ -140,6 +140,36 @@ describe('transcriptBlocksToDaemonMessages', () => { }); }); + it('preserves user input annotations metadata', () => { + const inputAnnotations = [ + { + type: 'reference' as const, + start: 0, + end: 14, + text: '@dataset:users', + reference: { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', + }, + }, + ]; + const messages = transcriptBlocksToDaemonMessages([ + textBlock('user-1', 'user', '@dataset:users', 1, false, { + meta: { inputAnnotations }, + }), + ]); + + expect(messages[0]).toMatchObject({ + id: 'user-1', + role: 'user', + content: '@dataset:users', + inputAnnotations, + }); + }); + it('hides background task notifications by metadata', () => { const messages = transcriptBlocksToDaemonMessages([ textBlock( diff --git a/packages/web-shell/client/adapters/transcriptToMessages.ts b/packages/web-shell/client/adapters/transcriptToMessages.ts index f847bbc0001..648ee9bc049 100644 --- a/packages/web-shell/client/adapters/transcriptToMessages.ts +++ b/packages/web-shell/client/adapters/transcriptToMessages.ts @@ -5,6 +5,7 @@ */ import type { + DaemonInputAnnotation, DaemonTranscriptBlock, DaemonTextTranscriptBlock, DaemonToolTranscriptBlock, @@ -36,6 +37,7 @@ type DaemonPermissionTranscriptBlock = Extract< type ExtendedDaemonTextTranscriptBlock = DaemonTextTranscriptBlock & { meta?: { source?: unknown; + inputAnnotations?: unknown; qwenDiscreteMessage?: boolean; backgroundTask?: unknown; }; @@ -224,12 +226,16 @@ export function transcriptBlocksToDaemonMessages( (textBlock as ExtendedDaemonTextTranscriptBlock).meta, ); const source = getString(meta, 'source'); + const inputAnnotations = Array.isArray(meta?.inputAnnotations) + ? (meta.inputAnnotations as DaemonInputAnnotation[]) + : undefined; const msg: DaemonUserMessage = { id: block.id, role: 'user', content: textBlock.text, timestamp: blockTime, ...(source ? { source } : {}), + ...(inputAnnotations ? { inputAnnotations } : {}), }; // Attach images if present if (textBlock.images && textBlock.images.length > 0) { diff --git a/packages/web-shell/client/components/ChatEditor.tsx b/packages/web-shell/client/components/ChatEditor.tsx index 26e1364aa87..af4d40c7973 100644 --- a/packages/web-shell/client/components/ChatEditor.tsx +++ b/packages/web-shell/client/components/ChatEditor.tsx @@ -27,6 +27,7 @@ import { } from '../customization'; import { useComposerCore, + type ComposerSubmitMetadata, type EditorHandle, type SlashMenuState, getComposerTagDisplay, @@ -35,7 +36,7 @@ import { } from '../hooks/useComposerCore'; import { AtMentionPanel } from './AtMentionPanel'; import { cssUrlVar } from '../utils/cssUrlVar'; -import { getComposerTagIconUrl } from './composerTagIcons'; +import { getComposerTagIconUrl } from '../utils/composerTag'; import { isSafeImageSrc } from './messages/Markdown'; import { ModeIcon } from './ModeIcon'; import { planSlashSectionRows } from '../utils/slashSectionPlan'; @@ -66,6 +67,7 @@ interface ChatEditorProps { text: string, images?: import('../adapters/promptTypes').PromptImage[], commitAccepted?: import('../hooks/useComposerCore').ComposerSubmitCommit, + metadata?: ComposerSubmitMetadata, ) => boolean | void; onCycleMode?: () => void; onToggleShortcuts?: () => void; diff --git a/packages/web-shell/client/components/ChatPane.test.tsx b/packages/web-shell/client/components/ChatPane.test.tsx index ddfb920e79d..4e5e9ef0e1d 100644 --- a/packages/web-shell/client/components/ChatPane.test.tsx +++ b/packages/web-shell/client/components/ChatPane.test.tsx @@ -17,7 +17,12 @@ let connectionState: any; let streamingStateValue: string; let pendingPermission: any; let latestOnSubmit: - | ((text: string, images?: unknown, commit?: () => void) => boolean) + | (( + text: string, + images?: unknown, + commit?: () => void, + metadata?: unknown, + ) => boolean) | undefined; let latestChatEditorProps: any; let latestFollowupAccept: ((suggestion: string) => void) | undefined; @@ -317,6 +322,28 @@ describe('ChatPane', () => { }); }); + it('forwards composer annotations with an idle prompt', () => { + const inputAnnotations = [ + { + start: 6, + end: 14, + text: '@.husky/', + type: 'file', + data: { path: '.husky/' }, + }, + ]; + render(); + act(() => { + latestOnSubmit!('check @.husky/', undefined, undefined, { + inputAnnotations, + }); + }); + expect(sendPrompt).toHaveBeenCalledWith('check @.husky/', { + inputAnnotations, + onAdmitted: expect.any(Function), + }); + }); + it('queues a prompt while the pane is already running', () => { streamingStateValue = 'responding'; render(); @@ -330,6 +357,32 @@ describe('ChatPane', () => { expect(sendPrompt).not.toHaveBeenCalled(); }); + it('forwards composer annotations with a queued prompt', () => { + streamingStateValue = 'responding'; + const inputAnnotations = [ + { + start: 6, + end: 14, + text: '@.husky/', + type: 'file', + data: { path: '.husky/' }, + }, + ]; + render(); + act(() => { + latestOnSubmit!('queue @.husky/', undefined, undefined, { + inputAnnotations, + }); + }); + expect(enqueuePrompt).toHaveBeenCalledWith( + 'queue @.husky/', + undefined, + undefined, + inputAnnotations, + ); + expect(sendPrompt).not.toHaveBeenCalled(); + }); + it('forwards images with a queued prompt', () => { streamingStateValue = 'responding'; const images = [{ data: 'image-data', media_type: 'image/png' }]; diff --git a/packages/web-shell/client/components/ChatPane.tsx b/packages/web-shell/client/components/ChatPane.tsx index d1a40d562e7..5d677f34ae4 100644 --- a/packages/web-shell/client/components/ChatPane.tsx +++ b/packages/web-shell/client/components/ChatPane.tsx @@ -19,6 +19,7 @@ import { extractPendingPermission } from '../adapters/transcriptAdapter'; import type { PromptImage } from '../adapters/promptTypes'; import type { ComposerSubmitCommit, + ComposerSubmitMetadata, EditorHandle, } from '../hooks/useComposerCore'; import { useQueuedPrompts } from '../hooks/useQueuedPrompts'; @@ -154,14 +155,17 @@ export function ChatPane({ title, onClose, onError }: ChatPaneProps) { text: string, images?: PromptImage[], commitAccepted?: ComposerSubmitCommit, + metadata?: ComposerSubmitMetadata, ): boolean => { const trimmed = text.trim(); if (!trimmed) return false; if (connection.status !== 'connected') return false; + const inputAnnotations = metadata?.inputAnnotations; if (streamingStateRef.current === 'idle') { actions .sendPrompt(trimmed, { ...(images && images.length ? { images } : {}), + ...(inputAnnotations ? { inputAnnotations } : {}), onAdmitted: () => { clearFollowup(); commitAccepted?.(); @@ -172,7 +176,9 @@ export function ChatPane({ title, onClose, onError }: ChatPaneProps) { ); return false; } - return enqueuePrompt(trimmed, images); + return inputAnnotations + ? enqueuePrompt(trimmed, images, undefined, inputAnnotations) + : enqueuePrompt(trimmed, images); }, [actions, clearFollowup, connection.status, enqueuePrompt, reportError], ); diff --git a/packages/web-shell/client/components/MessageItem.tsx b/packages/web-shell/client/components/MessageItem.tsx index 2ce849ddce6..0dce2bd68e7 100644 --- a/packages/web-shell/client/components/MessageItem.tsx +++ b/packages/web-shell/client/components/MessageItem.tsx @@ -57,6 +57,7 @@ export const MessageItem = memo(function MessageItem({ ); diff --git a/packages/web-shell/client/components/QueuedPromptDisplay.tsx b/packages/web-shell/client/components/QueuedPromptDisplay.tsx index 0b7c042bc8d..9587581923d 100644 --- a/packages/web-shell/client/components/QueuedPromptDisplay.tsx +++ b/packages/web-shell/client/components/QueuedPromptDisplay.tsx @@ -5,6 +5,7 @@ */ import type { PromptImage } from '../adapters/promptTypes'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; import deleteIconUrl from '../assets/icons/delete.svg'; import editIconUrl from '../assets/icons/edit.svg'; import insertIconUrl from '../assets/icons/insert.svg'; @@ -21,6 +22,7 @@ export interface QueuedPrompt { sessionId?: string; text: string; images?: PromptImage[]; + inputAnnotations?: DaemonInputAnnotation[]; onComplete?: () => void; serverPromptId?: string; serverState?: 'submitting' | 'queued' | 'running'; diff --git a/packages/web-shell/client/components/composerTagIcons.test.ts b/packages/web-shell/client/components/composerTagIcons.test.ts deleted file mode 100644 index a6af54304a3..00000000000 --- a/packages/web-shell/client/components/composerTagIcons.test.ts +++ /dev/null @@ -1,24 +0,0 @@ -import { describe, expect, it } from 'vitest'; -import { getComposerTagIconUrl } from './composerTagIcons'; - -describe('composer tag icons', () => { - it('uses registered icons for custom tag kinds', () => { - expect(getComposerTagIconUrl('table', { table: '/icons/table.svg' })).toBe( - '/icons/table.svg', - ); - }); - - it('falls back to built-in tag icons', () => { - expect(getComposerTagIconUrl('file')).toBeTruthy(); - }); - - it('ignores inherited object properties', () => { - const icons = Object.create({ table: '/icons/table.svg' }) as Record< - string, - string - >; - - expect(getComposerTagIconUrl('table', icons)).toBeUndefined(); - expect(getComposerTagIconUrl('toString')).toBeUndefined(); - }); -}); diff --git a/packages/web-shell/client/components/composerTagIcons.ts b/packages/web-shell/client/components/composerTagIcons.ts deleted file mode 100644 index c6225fd3230..00000000000 --- a/packages/web-shell/client/components/composerTagIcons.ts +++ /dev/null @@ -1,38 +0,0 @@ -import extensionIconUrl from '../assets/icons/at-extension.svg'; -import fileIconUrl from '../assets/icons/at-file.svg'; -import mcpIconUrl from '../assets/icons/at-mcp.svg'; -import skillIconUrl from '../assets/icons/at-skill.svg'; -import type { - WebShellBuiltinComposerTagKind, - WebShellComposerTagIconMap, - WebShellComposerTagKind, -} from '../customization'; - -const builtinTagIconUrls: Record = { - extension: extensionIconUrl, - file: fileIconUrl, - mcp: mcpIconUrl, - skill: skillIconUrl, -}; - -function getOwnIconUrl( - iconUrls: WebShellComposerTagIconMap | undefined, - kind: string, -): string | undefined { - if (!iconUrls || !Object.prototype.hasOwnProperty.call(iconUrls, kind)) { - return undefined; - } - const iconUrl = iconUrls[kind]; - return typeof iconUrl === 'string' ? iconUrl : undefined; -} - -export function getComposerTagIconUrl( - kind: WebShellComposerTagKind | undefined, - customIconUrls?: WebShellComposerTagIconMap, -): string | undefined { - if (!kind) return undefined; - return ( - getOwnIconUrl(customIconUrls, kind) ?? - getOwnIconUrl(builtinTagIconUrls, kind) - ); -} diff --git a/packages/web-shell/client/components/messages/UserMessage.test.tsx b/packages/web-shell/client/components/messages/UserMessage.test.tsx index 609638d6c0f..ff0ad4e22d6 100644 --- a/packages/web-shell/client/components/messages/UserMessage.test.tsx +++ b/packages/web-shell/client/components/messages/UserMessage.test.tsx @@ -29,12 +29,245 @@ function render(node: ReactNode): HTMLElement { return container; } +function referenceAnnotation( + content: string, + text: string, + reference: { + id: string; + kind?: string; + label?: string; + value?: string; + serialized?: string; + removable?: boolean; + }, +) { + const start = content.indexOf(text); + if (start < 0) { + throw new Error(`Missing annotation text: ${text}`); + } + return { + type: 'reference' as const, + start, + end: start + text.length, + text, + reference, + }; +} + +function findByTitle(container: HTMLElement, title: string): Element | null { + return ( + Array.from(container.querySelectorAll('[title]')).find( + (element) => element.getAttribute('title') === title, + ) ?? null + ); +} + describe('UserMessage', () => { it('renders content', () => { const container = render(); expect(container.textContent).toContain('hello world'); }); + it('renders file references as chips from input annotations', () => { + const content = 'list @.qwen/ files'; + const container = render( + , + ); + const chip = container.querySelector('[title="@.qwen/"]'); + + expect(chip).not.toBeNull(); + expect(chip?.textContent).toBe('.qwen/'); + expect(container.textContent).toContain('list .qwen/ files'); + }); + + it('uses configured composer tag icons for annotation chips', () => { + const content = 'list @.qwen/ files'; + const container = render( + + + , + ); + const icon = container.querySelector( + '[title="@.qwen/"] [aria-hidden="true"]', + ); + + expect(icon).not.toBeNull(); + expect(icon?.style.getPropertyValue('--user-message-tag-icon-url')).toBe( + 'url("/custom-file.svg")', + ); + }); + + it('renders extension and MCP references as chips from input annotations', () => { + const content = '@ext:browser and @mcp:docs'; + const container = render( + , + ); + + expect(container.querySelector('[title="@ext:browser"]')).not.toBeNull(); + expect(container.querySelector('[title="@mcp:docs"]')).not.toBeNull(); + expect(container.textContent).toContain('browser and docs'); + }); + + it('does not render inline email addresses as chips', () => { + const container = render(); + + expect(container.querySelector('[title="@b.test"]')).toBeNull(); + expect(container.textContent).toContain('a@b.test'); + }); + + it('keeps references as text without input annotations', () => { + const container = render(); + + expect(container.querySelector('[title="@dataset:users"]')).toBeNull(); + expect(container.textContent).toContain('open @dataset:users'); + }); + + it('renders extensionless file references from input annotations', () => { + const content = 'open @Makefile and @src/Makefile'; + const container = render( + , + ); + + expect(container.querySelector('[title="@Makefile"]')).not.toBeNull(); + expect(container.querySelector('[title="@src/Makefile"]')).not.toBeNull(); + expect(container.textContent).toContain('open Makefile and src/Makefile'); + }); + + it('keeps MCP resource trailing punctuation from input annotations', () => { + const serialized = '@docs\\:res\\://doc.'; + const content = `open ${serialized} now`; + const container = render( + , + ); + + expect(findByTitle(container, serialized)).not.toBeNull(); + expect(container.textContent).toContain('open docs:res://doc. now'); + }); + + it('keeps escaped trailing punctuation from input annotations', () => { + const serialized = '@path\\:'; + const content = `open ${serialized}`; + const container = render( + , + ); + + expect(findByTitle(container, serialized)).not.toBeNull(); + expect(container.textContent).toContain('open path:'); + }); + + it('renders custom provider references from input annotations', () => { + const container = render( + + + , + ); + const chip = container.querySelector('[title="@dataset:users"]'); + const icon = chip?.querySelector('[aria-hidden="true"]'); + + expect(chip).not.toBeNull(); + expect(chip?.textContent).toBe('Datasetusers'); + expect(icon?.style.getPropertyValue('--user-message-tag-icon-url')).toBe( + 'url("/dataset.svg")', + ); + }); + it('renders images when provided', () => { const container = render( { const container = render( ( - {content.slice(1)} + renderUserMessageContent: ({ content, inputAnnotations }) => ( + + {content.slice(1)}:{inputAnnotations?.length ?? 0} + ), }} > - + , ); expect(container.querySelector('[data-testid="tag-chip"]')).not.toBeNull(); - expect(container.textContent).toContain('.husky/_/husky.sh xuyao'); + expect(container.textContent).toContain('.husky/_/husky.sh xuyao:1'); }); it('renders parsed user-message tag parts', () => { diff --git a/packages/web-shell/client/components/messages/UserMessage.tsx b/packages/web-shell/client/components/messages/UserMessage.tsx index 70278976b82..41c18c70050 100644 --- a/packages/web-shell/client/components/messages/UserMessage.tsx +++ b/packages/web-shell/client/components/messages/UserMessage.tsx @@ -1,13 +1,20 @@ import { + Fragment, memo, - useMemo, useCallback, useEffect, useLayoutEffect, + useMemo, useRef, useState, type ReactNode, } from 'react'; +import { + getComposerTagIconUrl, + getComposerTagViewModel, + splitComposerTagContentByAnnotations, +} from '../../utils/composerTag'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; import { isSafeImageSrc } from './Markdown'; import { useWebShellCustomization } from '../../customization'; import type { @@ -24,7 +31,6 @@ import { } from '../../hooks/useComposerCore'; import { useI18n } from '../../i18n'; import { cssUrlVar } from '../../utils/cssUrlVar'; -import { getComposerTagIconUrl } from '../composerTagIcons'; import flashStyles from '../MessageLocateFlash.module.css'; import styles from './UserMessage.module.css'; @@ -36,12 +42,57 @@ interface UserMessageImage { interface UserMessageProps { content: string; images?: UserMessageImage[]; + inputAnnotations?: readonly DaemonInputAnnotation[]; isLocateFlashing?: boolean; } +function DefaultUserMessageContent({ + composerTagIcons, + content, + inputAnnotations, + onComposerTagClick, + renderComposerTag, + renderComposerTagTooltip, +}: { + composerTagIcons?: WebShellComposerTagIconMap; + content: string; + inputAnnotations?: readonly DaemonInputAnnotation[]; + onComposerTagClick?: ComposerTagClickHandler; + renderComposerTag?: ComposerTagRenderer; + renderComposerTagTooltip?: ComposerTagRenderer; +}) { + // Submit-time annotations are the source of truth for reference chips. + // Unannotated serialized text stays plain text. + const segments = useMemo( + () => splitComposerTagContentByAnnotations(content, inputAnnotations), + [content, inputAnnotations], + ); + return ( + <> + {segments.map((segment, index) => + segment.type === 'text' ? ( + {segment.text} + ) : ( + + ), + )} + + ); +} + export const UserMessage = memo(function UserMessage({ content, images, + inputAnnotations, isLocateFlashing = false, }: UserMessageProps) { const { t } = useI18n(); @@ -57,8 +108,24 @@ export const UserMessage = memo(function UserMessage({ const [expanded, setExpanded] = useState(false); const [heightOverflowing, setHeightOverflowing] = useState(false); const renderedContent = useMemo(() => { - const explicit = renderUserMessageContent?.({ content, images }); + const explicit = renderUserMessageContent?.({ + content, + images, + inputAnnotations, + }); if (explicit !== undefined && explicit !== null) return explicit; + if (inputAnnotations && inputAnnotations.length > 0) { + return ( + + ); + } let parts: readonly WebShellUserMessagePart[] | undefined | null; try { parts = parseUserMessageContent?.(content); @@ -83,6 +150,7 @@ export const UserMessage = memo(function UserMessage({ }, [ content, images, + inputAnnotations, onComposerTagClick, parseUserMessageContent, composerTagIcons, @@ -185,12 +253,16 @@ function UserMessageTag({ renderComposerTag, renderComposerTagTooltip, onComposerTagClick, + title, + preserveCustomKindLabel = false, }: { tag: WebShellComposerTag; composerTagIcons: WebShellComposerTagIconMap | undefined; renderComposerTag: ComposerTagRenderer | undefined; renderComposerTagTooltip: ComposerTagRenderer | undefined; onComposerTagClick: ComposerTagClickHandler | undefined; + title?: string; + preserveCustomKindLabel?: boolean; }) { const info = { tag, placement: 'user-message' as const, readonly: true }; let custom: ReactNode | null | undefined; @@ -206,10 +278,17 @@ function UserMessageTag({ console.warn('[WebShell] user message tag tooltip render failed', error); } const clickable = Boolean(onComposerTagClick); + const viewModel = preserveCustomKindLabel + ? getComposerTagViewModel(tag, composerTagIcons) + : null; const rawTagLabel = getComposerTagLabel(tag); - const tagValue = getComposerTagValue(tag); - const tagLabel = tag.kind ? '' : rawTagLabel; - const iconUrl = tag.icon ?? getComposerTagIconUrl(tag.kind, composerTagIcons); + const tagValue = viewModel?.tagValue ?? getComposerTagValue(tag); + const tagLabel = viewModel?.tagLabel ?? (tag.kind ? '' : rawTagLabel); + const fallback = viewModel?.fallback ?? tag.id; + const iconUrl = + tag.icon ?? + viewModel?.iconUrl ?? + getComposerTagIconUrl(tag.kind, composerTagIcons); const safeIconUrl = iconUrl && isSafeImageSrc(iconUrl) ? iconUrl : undefined; return ( { if (!clickable) return; event.stopPropagation(); @@ -252,7 +331,7 @@ function UserMessageTag({ {tagValue ? ( {tagValue} ) : !tagLabel ? ( - {tag.id} + {fallback} ) : null} )} diff --git a/packages/web-shell/client/customization.tsx b/packages/web-shell/client/customization.tsx index 9f8f2f0617d..fc282d8f84c 100644 --- a/packages/web-shell/client/customization.tsx +++ b/packages/web-shell/client/customization.tsx @@ -5,6 +5,7 @@ import { type ReactNode, } from 'react'; import type { Components, Options } from 'react-markdown'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; import type { DaemonStreamingState } from '@qwen-code/webui/daemon-react-sdk'; import type { ACPToolCall } from './adapters/types'; import type { WelcomeHeaderProps } from './components/WelcomeHeader'; @@ -93,6 +94,7 @@ export type WelcomeFooterRenderer = (props: WelcomeHeaderProps) => ReactNode; export interface UserMessageContentRenderInfo { content: string; images?: readonly { data: string; mimeType: string }[]; + inputAnnotations?: readonly DaemonInputAnnotation[]; } export type UserMessageContentRenderer = ( diff --git a/packages/web-shell/client/hooks/useComposerCore.dom.test.tsx b/packages/web-shell/client/hooks/useComposerCore.dom.test.tsx index 2a7b77c78fc..bb787a79e08 100644 --- a/packages/web-shell/client/hooks/useComposerCore.dom.test.tsx +++ b/packages/web-shell/client/hooks/useComposerCore.dom.test.tsx @@ -160,6 +160,21 @@ describe('useComposerCore inline tags', () => { ' explain', undefined, expect.any(Function), + { + inputAnnotations: [ + { + end: 9, + reference: { + id: 'orders', + serialized: '
', + value: 'orders', + }, + start: 0, + text: '
', + type: 'reference', + }, + ], + }, ); }); }); diff --git a/packages/web-shell/client/hooks/useComposerCore.ts b/packages/web-shell/client/hooks/useComposerCore.ts index 155e51951b1..8a2aa862848 100644 --- a/packages/web-shell/client/hooks/useComposerCore.ts +++ b/packages/web-shell/client/hooks/useComposerCore.ts @@ -60,7 +60,12 @@ import { } from '../extensions/inputHighlight'; import { isEditableTarget } from '../utils/dom'; import { cssUrlValue } from '../utils/cssUrlVar'; -import { getComposerTagIconUrl } from '../components/composerTagIcons'; +import { + createInputAnnotationsFromComposerTags, + getComposerTagIconUrl, + getComposerTagSerialized, +} from '../utils/composerTag'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; import { isSafeImageSrc } from '../components/messages/Markdown'; import type { ComposerTagClickHandler, @@ -488,9 +493,7 @@ export function expandLargePastePlaceholders( // ---- Tag serialization (shared) ---- export function serializeComposerTag(tag: WebShellComposerTag): string { - return ( - tag.serialized?.trim() || tag.value?.trim() || tag.label?.trim() || tag.id - ); + return getComposerTagSerialized(tag); } function serializeComposerTags(tags: readonly WebShellComposerTag[]): string { @@ -881,12 +884,11 @@ const inlineComposerTagField = StateField.define({ export function getInlineComposerTags(view: EditorView): WebShellComposerTag[] { const tags: WebShellComposerTag[] = []; - view.state - .field(inlineComposerTagField) - .between(0, view.state.doc.length, (_from, _to, value) => { - const tag = (value.spec as Partial).tag; - if (tag) tags.push(toPublicComposerTag(tag)); - }); + const inlineTags = view.state.field(inlineComposerTagField, false); + inlineTags?.between(0, view.state.doc.length, (_from, _to, value) => { + const tag = (value.spec as Partial).tag; + if (tag) tags.push(toPublicComposerTag(tag)); + }); return tags; } @@ -1011,11 +1013,16 @@ function createFollowupGhostExtension(suggestion: string | null) { export type ComposerSubmitCommit = () => void; +export interface ComposerSubmitMetadata { + inputAnnotations?: DaemonInputAnnotation[]; +} + export interface UseComposerCoreOptions { onSubmit: ( text: string, images?: PromptImage[], commitAccepted?: ComposerSubmitCommit, + metadata?: ComposerSubmitMetadata, ) => boolean | void; onCycleMode?: () => void; onToggleShortcuts?: () => void; @@ -1732,7 +1739,7 @@ export function useComposerCore( })) : []; const tags = tagsOverride ?? composerTagsRef.current; - if (!rawText && tags.length === 0) return true; + if (!rawText && tags.length === 0 && inlineTags.length === 0) return true; const textWithInlineTags = tagsOverride === undefined ? replaceInlineTagPlacements(rawText, normalizedInlineTags) @@ -1744,6 +1751,11 @@ export function useComposerCore( const prompt = buildComposerPrompt(text, tags); const images = pastedImagesRef.current; const isShellMode = shellModeRef.current; + const promptText = isShellMode ? `!${prompt}` : prompt; + const inputAnnotations = createInputAnnotationsFromComposerTags( + promptText, + [...tags, ...normalizedInlineTags.map((placement) => placement.tag)], + ); let committed = false; const commitAccepted = () => { if (committed) return; @@ -1771,9 +1783,10 @@ export function useComposerCore( }); }; const accepted = onSubmitRef.current( - isShellMode ? `!${prompt}` : prompt, + promptText, images.length > 0 ? [...images] : undefined, commitAccepted, + inputAnnotations.length > 0 ? { inputAnnotations } : undefined, ); if (accepted === false) return true; commitAccepted(); @@ -2819,11 +2832,7 @@ export function useComposerCore( if (!view) return; const inlineTags = getInlineComposerTags(view); if (input?.tagPlacement === 'inline') { - submitTextRef.current( - view, - buildComposerPrompt(input.text ?? '', input.tags ?? inlineTags), - [], - ); + submitTextRef.current(view, input.text ?? '', input.tags ?? inlineTags); return; } if ( @@ -2831,11 +2840,7 @@ export function useComposerCore( input.tags === undefined && inlineTags.length > 0 ) { - submitTextRef.current( - view, - buildComposerPrompt(input.text, inlineTags), - [], - ); + submitTextRef.current(view, input.text, inlineTags); return; } submitTextRef.current( diff --git a/packages/web-shell/client/hooks/useQueuedPrompts.ts b/packages/web-shell/client/hooks/useQueuedPrompts.ts index 08c239727ad..df98c6a623b 100644 --- a/packages/web-shell/client/hooks/useQueuedPrompts.ts +++ b/packages/web-shell/client/hooks/useQueuedPrompts.ts @@ -23,6 +23,7 @@ import { type DaemonStreamingState, } from '@qwen-code/webui/daemon-react-sdk'; import type { + DaemonInputAnnotation, DaemonPendingPromptSummary, DaemonTranscriptStore, } from '@qwen-code/sdk/daemon'; @@ -74,7 +75,9 @@ function areQueuedPromptsEqual( prompt.serverState === other.serverState && prompt.isEditing === other.isEditing && prompt.isRemoving === other.isRemoving && - (prompt.images?.length ?? 0) === (other.images?.length ?? 0) + (prompt.images?.length ?? 0) === (other.images?.length ?? 0) && + (prompt.inputAnnotations?.length ?? 0) === + (other.inputAnnotations?.length ?? 0) ); }); } @@ -96,6 +99,7 @@ export interface UseQueuedPromptsResult { text: string, images?: PromptImage[], onComplete?: () => void, + inputAnnotations?: DaemonInputAnnotation[], ) => boolean; removeQueuedPrompt: (id: number) => void; insertQueuedPrompt: (id: number) => Promise; @@ -359,7 +363,13 @@ export function useQueuedPrompts({ const text = prompt?.text ?? ''; if (text) { displayedServerPromptIdsRef.current.add(promptId); - store.appendLocalUserMessage(text, toStoreImages(prompt?.images)); + store.appendLocalUserMessage( + text, + toStoreImages(prompt?.images), + prompt?.inputAnnotations?.length + ? { inputAnnotations: prompt.inputAnnotations } + : undefined, + ); } } void refreshPendingPrompts(); @@ -415,7 +425,12 @@ export function useQueuedPrompts({ ); const enqueuePrompt = useCallback( - (text: string, images?: PromptImage[], onComplete?: () => void) => { + ( + text: string, + images?: PromptImage[], + onComplete?: () => void, + inputAnnotations?: DaemonInputAnnotation[], + ) => { const trimmed = text.trim(); if (!trimmed) return true; const localId = nextQueuedPromptIdRef.current++; @@ -423,11 +438,15 @@ export function useQueuedPrompts({ const submitAbort = new AbortController(); submitAbortControllersRef.current.add(submitAbort); const queuedImages = images ? [...images] : undefined; + const queuedAnnotations = inputAnnotations + ? [...inputAnnotations] + : undefined; const nextPrompt: QueuedPrompt = { id: localId, sessionId: targetSessionId, text: trimmed, images: queuedImages, + inputAnnotations: queuedAnnotations, onComplete, serverState: 'submitting', }; @@ -437,6 +456,7 @@ export function useQueuedPrompts({ sessionActions .submitPrompt(trimmed, { images, + inputAnnotations: queuedAnnotations, optimisticUserMessage: false, sessionId: targetSessionId, signal: submitAbort.signal, @@ -462,6 +482,9 @@ export function useQueuedPrompts({ store.appendLocalUserMessage( trimmed, toStoreImages(queuedImages), + queuedAnnotations?.length + ? { inputAnnotations: queuedAnnotations } + : undefined, ); } const next = queuedPromptsRef.current.filter( diff --git a/packages/web-shell/client/utils/composerTag.test.ts b/packages/web-shell/client/utils/composerTag.test.ts new file mode 100644 index 00000000000..396ef3f005e --- /dev/null +++ b/packages/web-shell/client/utils/composerTag.test.ts @@ -0,0 +1,511 @@ +import { describe, expect, it } from 'vitest'; +import { + createInputAnnotationsFromComposerTags, + getComposerTagIconUrl, + getComposerTagViewModel, + splitComposerTagContentByAnnotations, +} from './composerTag'; + +function referenceAnnotation( + content: string, + text: string, + reference: { + id: string; + kind?: string; + label?: string; + value?: string; + serialized?: string; + removable?: boolean; + }, +) { + const start = content.indexOf(text); + if (start < 0) { + throw new Error(`Missing annotation text: ${text}`); + } + return { + type: 'reference' as const, + start, + end: start + text.length, + text, + reference, + }; +} + +describe('composer tag icon URLs', () => { + it('uses registered icons for custom tag kinds', () => { + expect(getComposerTagIconUrl('table', { table: '/icons/table.svg' })).toBe( + '/icons/table.svg', + ); + }); + + it('falls back to built-in tag icons', () => { + expect(getComposerTagIconUrl('file')).toBeTruthy(); + }); + + it('ignores inherited object properties', () => { + const icons = Object.create({ table: '/icons/table.svg' }) as Record< + string, + string + >; + + expect(getComposerTagIconUrl('table', icons)).toBeUndefined(); + expect(getComposerTagIconUrl('toString')).toBeUndefined(); + }); +}); + +describe('getComposerTagViewModel', () => { + it('returns display fields for custom tags', () => { + expect( + getComposerTagViewModel({ + id: 'custom:1', + label: ' Dataset ', + value: ' users.csv ', + }), + ).toEqual({ + tagLabel: 'Dataset', + tagValue: 'users.csv', + fallback: 'custom:1', + iconUrl: undefined, + }); + }); + + it('hides labels for built-in tag kinds and resolves icons', () => { + const model = getComposerTagViewModel({ + id: 'file:@.qwen/', + kind: 'file', + label: 'File', + value: '.qwen/', + }); + + expect(model.tagLabel).toBe(''); + expect(model.tagValue).toBe('.qwen/'); + expect(model.fallback).toBe('file:@.qwen/'); + expect(model.iconUrl).toBeTruthy(); + }); + + it('uses custom icon maps when provided', () => { + expect( + getComposerTagViewModel( + { + id: 'file:@src/index.ts', + kind: 'file', + value: 'src/index.ts', + }, + { file: '/custom-file.svg' }, + ).iconUrl, + ).toBe('/custom-file.svg'); + }); + + it('uses fallback when tag has no display text', () => { + expect(getComposerTagViewModel({ id: 'tag-id' })).toEqual({ + tagLabel: '', + tagValue: '', + fallback: 'tag-id', + iconUrl: undefined, + }); + }); + + it('keeps labels for custom tag kinds', () => { + expect( + getComposerTagViewModel({ + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + }), + ).toEqual({ + tagLabel: 'Dataset', + tagValue: 'users', + fallback: 'dataset:users', + iconUrl: undefined, + }); + }); +}); + +describe('composer tag input annotations', () => { + it('creates reference annotations using ranges from final prompt text', () => { + expect( + createInputAnnotationsFromComposerTags('@dataset:users\n\nshow rows', [ + { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', + }, + ]), + ).toEqual([ + { + type: 'reference', + start: 0, + end: 14, + text: '@dataset:users', + reference: { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', + }, + }, + ]); + }); + + it('creates annotations for extensionless file references', () => { + expect( + createInputAnnotationsFromComposerTags( + '@Makefile @LICENSE @src/Makefile', + [ + { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }, + { + id: 'file:@LICENSE', + kind: 'file', + value: 'LICENSE', + serialized: '@LICENSE', + }, + { + id: 'file:@src/Makefile', + kind: 'file', + value: 'src/Makefile', + serialized: '@src/Makefile', + }, + ], + ), + ).toEqual([ + { + type: 'reference', + start: 0, + end: 9, + text: '@Makefile', + reference: { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }, + }, + { + type: 'reference', + start: 10, + end: 18, + text: '@LICENSE', + reference: { + id: 'file:@LICENSE', + kind: 'file', + value: 'LICENSE', + serialized: '@LICENSE', + }, + }, + { + type: 'reference', + start: 19, + end: 32, + text: '@src/Makefile', + reference: { + id: 'file:@src/Makefile', + kind: 'file', + value: 'src/Makefile', + serialized: '@src/Makefile', + }, + }, + ]); + }); + + it('returns no annotations for empty tags', () => { + expect( + createInputAnnotationsFromComposerTags('show @Makefile', []), + ).toEqual([]); + }); + + it('skips tags whose serialized text is absent from the prompt', () => { + expect( + createInputAnnotationsFromComposerTags('show @Makefile', [ + { + id: 'file:@LICENSE', + kind: 'file', + value: 'LICENSE', + serialized: '@LICENSE', + }, + { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }, + ]), + ).toEqual([ + { + type: 'reference', + start: 5, + end: 14, + text: '@Makefile', + reference: { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }, + }, + ]); + }); + + it('matches repeated serialized references in order', () => { + expect( + createInputAnnotationsFromComposerTags('@file @file', [ + { + id: 'file:first', + kind: 'file', + value: 'first', + serialized: '@file', + }, + { + id: 'file:second', + kind: 'file', + value: 'second', + serialized: '@file', + }, + ]), + ).toEqual([ + { + type: 'reference', + start: 0, + end: 5, + text: '@file', + reference: { + id: 'file:first', + kind: 'file', + value: 'first', + serialized: '@file', + }, + }, + { + type: 'reference', + start: 6, + end: 11, + text: '@file', + reference: { + id: 'file:second', + kind: 'file', + value: 'second', + serialized: '@file', + }, + }, + ]); + }); + + it('uses annotations for custom provider references', () => { + expect( + splitComposerTagContentByAnnotations('open @dataset:users now', [ + { + type: 'reference', + start: 5, + end: 19, + text: '@dataset:users', + reference: { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', + }, + }, + ]), + ).toEqual([ + { type: 'text', text: 'open ' }, + { + type: 'reference', + tag: { + id: 'dataset:users', + kind: 'dataset', + label: 'Dataset', + value: 'users', + serialized: '@dataset:users', + }, + }, + { type: 'text', text: ' now' }, + ]); + }); + + it('uses annotations for extensionless file references', () => { + const content = 'open @Makefile and @src/Makefile'; + + expect( + splitComposerTagContentByAnnotations(content, [ + referenceAnnotation(content, '@Makefile', { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }), + referenceAnnotation(content, '@src/Makefile', { + id: 'file:@src/Makefile', + kind: 'file', + value: 'src/Makefile', + serialized: '@src/Makefile', + }), + ]), + ).toEqual([ + { type: 'text', text: 'open ' }, + { + type: 'reference', + tag: { + id: 'file:@Makefile', + kind: 'file', + value: 'Makefile', + serialized: '@Makefile', + }, + }, + { type: 'text', text: ' and ' }, + { + type: 'reference', + tag: { + id: 'file:@src/Makefile', + kind: 'file', + value: 'src/Makefile', + serialized: '@src/Makefile', + }, + }, + ]); + }); + + it('keeps MCP resource trailing punctuation from annotations', () => { + const serialized = '@docs\\:res\\://doc.'; + const content = `open ${serialized} now`; + + expect( + splitComposerTagContentByAnnotations(content, [ + referenceAnnotation(content, serialized, { + id: `mcp:${serialized}`, + kind: 'mcp', + value: 'docs:res://doc.', + serialized, + }), + ]), + ).toEqual([ + { type: 'text', text: 'open ' }, + { + type: 'reference', + tag: { + id: 'mcp:@docs\\:res\\://doc.', + kind: 'mcp', + value: 'docs:res://doc.', + serialized: '@docs\\:res\\://doc.', + }, + }, + { type: 'text', text: ' now' }, + ]); + }); + + it('keeps escaped trailing punctuation from annotations', () => { + const serialized = '@path\\:'; + const content = `open ${serialized}`; + + expect( + splitComposerTagContentByAnnotations(content, [ + referenceAnnotation(content, serialized, { + id: `file:${serialized}`, + kind: 'file', + value: 'path:', + serialized, + }), + ]), + ).toEqual([ + { type: 'text', text: 'open ' }, + { + type: 'reference', + tag: { + id: 'file:@path\\:', + kind: 'file', + value: 'path:', + serialized: '@path\\:', + }, + }, + ]); + }); + + it('leaves unannotated references as text', () => { + expect(splitComposerTagContentByAnnotations('list @.qwen/ files')).toEqual([ + { type: 'text', text: 'list @.qwen/ files' }, + ]); + }); + + it('leaves invalid annotation ranges as text', () => { + expect( + splitComposerTagContentByAnnotations('list @.qwen/ files', [ + { + type: 'reference', + start: 5, + end: 12, + text: '@wrong/', + reference: { + id: 'file:@wrong/', + kind: 'file', + value: 'wrong/', + serialized: '@wrong/', + }, + }, + ]), + ).toEqual([{ type: 'text', text: 'list @.qwen/ files' }]); + }); + + it('leaves malformed reference annotations as text', () => { + expect( + splitComposerTagContentByAnnotations('list @.qwen/ files', [ + { + type: 'reference', + start: 5, + end: 12, + text: '@.qwen/', + } as unknown as DaemonInputAnnotation, + ]), + ).toEqual([{ type: 'text', text: 'list @.qwen/ files' }]); + }); + + it('skips overlapping annotations', () => { + expect( + splitComposerTagContentByAnnotations('open @one @two', [ + { + type: 'reference', + start: 5, + end: 9, + text: '@one', + reference: { + id: 'file:@one', + kind: 'file', + value: 'one', + serialized: '@one', + }, + }, + { + type: 'reference', + start: 8, + end: 13, + text: 'e @tw', + reference: { + id: 'file:overlap', + kind: 'file', + value: 'overlap', + serialized: 'e @tw', + }, + }, + ]), + ).toEqual([ + { type: 'text', text: 'open ' }, + { + type: 'reference', + tag: { + id: 'file:@one', + kind: 'file', + value: 'one', + serialized: '@one', + }, + }, + { type: 'text', text: ' @two' }, + ]); + }); +}); diff --git a/packages/web-shell/client/utils/composerTag.ts b/packages/web-shell/client/utils/composerTag.ts new file mode 100644 index 00000000000..41a037c5554 --- /dev/null +++ b/packages/web-shell/client/utils/composerTag.ts @@ -0,0 +1,158 @@ +import extensionIconUrl from '../assets/icons/at-extension.svg'; +import fileIconUrl from '../assets/icons/at-file.svg'; +import mcpIconUrl from '../assets/icons/at-mcp.svg'; +import skillIconUrl from '../assets/icons/at-skill.svg'; +import type { DaemonInputAnnotation } from '@qwen-code/sdk/daemon'; +import type { + WebShellBuiltinComposerTagKind, + WebShellComposerTag, + WebShellComposerTagIconMap, + WebShellComposerTagKind, +} from '../customization'; + +// Shared UI-facing shape for composer tags across React and CodeMirror. +export interface ComposerTagViewModel { + tagLabel: string; + tagValue: string; + fallback: string; + iconUrl?: string; +} + +export type ComposerTagContentSegment = + | { type: 'text'; text: string } + | { type: 'reference'; tag: WebShellComposerTag }; + +// Resolves tag kind metadata to asset URLs; it does not render icon components. +const builtinTagIconUrls: Record = { + extension: extensionIconUrl, + file: fileIconUrl, + mcp: mcpIconUrl, + skill: skillIconUrl, +}; + +function getOwnIconUrl( + iconUrls: WebShellComposerTagIconMap | undefined, + kind: string, +): string | undefined { + if (!iconUrls || !Object.prototype.hasOwnProperty.call(iconUrls, kind)) { + return undefined; + } + const iconUrl = iconUrls[kind]; + return typeof iconUrl === 'string' ? iconUrl : undefined; +} + +export function getComposerTagSerialized(tag: WebShellComposerTag): string { + return ( + tag.serialized?.trim() || tag.value?.trim() || tag.label?.trim() || tag.id + ); +} + +export function getComposerTagIconUrl( + kind: WebShellComposerTagKind | undefined, + customIconUrls?: WebShellComposerTagIconMap, +): string | undefined { + if (!kind) return undefined; + return ( + getOwnIconUrl(customIconUrls, kind) ?? + getOwnIconUrl(builtinTagIconUrls, kind) + ); +} + +export function createInputAnnotationsFromComposerTags( + content: string, + tags: readonly WebShellComposerTag[], +): DaemonInputAnnotation[] { + const annotations: DaemonInputAnnotation[] = []; + let cursor = 0; + for (const tag of tags) { + const serialized = getComposerTagSerialized(tag); + if (!serialized) continue; + const start = content.indexOf(serialized, cursor); + if (start < 0) continue; + const end = start + serialized.length; + annotations.push({ + type: 'reference', + start, + end, + text: serialized, + reference: { + id: tag.id, + ...(tag.kind ? { kind: tag.kind } : {}), + ...(tag.label ? { label: tag.label } : {}), + ...(tag.value ? { value: tag.value } : {}), + ...(tag.serialized ? { serialized: tag.serialized } : {}), + ...(tag.removable !== undefined ? { removable: tag.removable } : {}), + }, + }); + cursor = end; + } + return annotations; +} + +// User messages render chips only from submit-time annotations. Without that +// metadata, the serialized text remains plain text instead of being guessed. +export function splitComposerTagContentByAnnotations( + content: string, + inputAnnotations?: readonly DaemonInputAnnotation[], +): ComposerTagContentSegment[] { + if (!inputAnnotations || inputAnnotations.length === 0) { + return [{ type: 'text', text: content }]; + } + const segments: ComposerTagContentSegment[] = []; + let cursor = 0; + for (const annotation of inputAnnotations) { + if (annotation.type !== 'reference') continue; + const { start, end, text } = annotation; + const reference: DaemonInputAnnotation['reference'] | undefined = + annotation.reference; + if ( + !reference || + typeof reference.id !== 'string' || + start < cursor || + end <= start || + end > content.length || + content.slice(start, end) !== text + ) { + continue; + } + if (cursor < start) { + segments.push({ type: 'text', text: content.slice(cursor, start) }); + } + segments.push({ + type: 'reference', + tag: { + id: reference.id, + ...(reference.kind ? { kind: reference.kind } : {}), + ...(reference.label ? { label: reference.label } : {}), + ...(reference.value ? { value: reference.value } : {}), + serialized: reference.serialized ?? text, + ...(reference.removable !== undefined + ? { removable: reference.removable } + : {}), + }, + }); + cursor = end; + } + if (cursor < content.length) { + segments.push({ type: 'text', text: content.slice(cursor) }); + } + return segments.length > 0 ? segments : [{ type: 'text', text: content }]; +} + +// Normalizes display fields so each renderer can keep its own shell and styles. +export function getComposerTagViewModel( + tag: WebShellComposerTag, + composerTagIcons?: WebShellComposerTagIconMap, +): ComposerTagViewModel { + const rawTagLabel = tag.label?.trim() ?? ''; + const tagValue = tag.value?.trim() ?? ''; + const isBuiltinTag = tag.kind + ? getOwnIconUrl(builtinTagIconUrls, tag.kind) + : undefined; + return { + tagLabel: isBuiltinTag ? '' : rawTagLabel, + tagValue, + fallback: tag.id, + iconUrl: getComposerTagIconUrl(tag.kind, composerTagIcons), + }; +} diff --git a/packages/webui/src/daemon/session/actions.ts b/packages/webui/src/daemon/session/actions.ts index ee6e79e3259..040fc52a657 100644 --- a/packages/webui/src/daemon/session/actions.ts +++ b/packages/webui/src/daemon/session/actions.ts @@ -287,12 +287,23 @@ export function createDaemonSessionActions({ mimeType: img.mimeType || img.mediaType || img.media_type || 'image/*', })); + const inputAnnotations = + options?.inputAnnotations && options.inputAnnotations.length > 0 + ? options.inputAnnotations + : undefined; if (options?.optimisticUserMessage !== false) { - store.appendLocalUserMessage(text, normalizedImages); + store.appendLocalUserMessage( + text, + normalizedImages, + inputAnnotations ? { inputAnnotations } : undefined, + ); } const promptRequest: Record = { prompt: toDaemonPromptContent(text, normalizedImages), }; + if (inputAnnotations) { + promptRequest['_meta'] = { inputAnnotations }; + } if (options?.retry) { promptRequest['retry'] = true; } @@ -359,12 +370,23 @@ export function createDaemonSessionActions({ data: img.data, mimeType: img.mimeType || img.mediaType || img.media_type || 'image/*', })); + const inputAnnotations = + options?.inputAnnotations && options.inputAnnotations.length > 0 + ? options.inputAnnotations + : undefined; if (options?.optimisticUserMessage !== false) { - store.appendLocalUserMessage(text, normalizedImages); + store.appendLocalUserMessage( + text, + normalizedImages, + inputAnnotations ? { inputAnnotations } : undefined, + ); } const promptRequest: Record = { prompt: toDaemonPromptContent(text, normalizedImages), }; + if (inputAnnotations) { + promptRequest['_meta'] = { inputAnnotations }; + } if (options?.retry) { promptRequest['retry'] = true; } diff --git a/packages/webui/src/daemon/session/types.ts b/packages/webui/src/daemon/session/types.ts index 2bb47d12d2c..62252371960 100644 --- a/packages/webui/src/daemon/session/types.ts +++ b/packages/webui/src/daemon/session/types.ts @@ -12,6 +12,7 @@ import type { DaemonApprovalModeResult, DaemonAvailableCommand, DaemonForkSessionResult, + DaemonInputAnnotation, DaemonSessionBtwResult, DaemonMidTurnMessageResult, DaemonPendingPromptsResult, @@ -232,6 +233,7 @@ export interface DaemonCommandInfo { export interface SendPromptOptions { optimisticUserMessage?: boolean; images?: DaemonPromptImage[]; + inputAnnotations?: DaemonInputAnnotation[]; /** * When true, the daemon strips orphaned user entries from the chat * history before re-sending, and skips recording a duplicate user