feat(web-shell): add native Live Voice - #7859
Conversation
…voice # Conflicts: # packages/web-shell/client/components/ChatEditor.tsx
🖼️ web-shell visual previewRendered against a mock daemon (no real backend): the PR base vs this PR head Screenshots · before / afterℹ️ No screenshot changed against the PR base — but this PR edits 10 render-shaping files:
Either the change has no visual effect (logic, plumbing, a state the scenarios never reach), or no scenario renders this UI — in which case the preview cannot see it, and an empty result is a coverage gap rather than a clean bill of health. To make it visible, add a scenario to Full-resolution recordings (.webm) are attached to the workflow run. — Qwen Code · web-shell visuals |
🩺 serve daemon A/BBuilt the PR base vs this PR head ✅ No response changes against the PR base across 4 scenario(s). — Qwen Code · serve A/B |
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
@LaZzyMan thanks for putting this together — the design doc, test plan, and fail-closed posture are all well thought out, and the PR description is one of the more thorough ones we've seen.
That said, this is a 21,400-line, 111-file feature spanning five packages (core, cli, desktop, sdk-typescript, acp-bridge), a new Electron + Swift native host, a realtime voice provider, and system-level macOS permissions (Accessibility, Screen Recording, Input Monitoring). It also reuses DashScope credentials for the realtime provider and introduces new computer-use trust boundaries (Appshot gating, hostile on-screen instruction handling). Those are exactly the areas — auth, credential flow, and computer-use trust — where we need a maintainer to weigh in on direction before anyone dives into a code review of this scale.
A few things worth discussing before this moves forward:
- Scope. Could this land as a sequence of smaller PRs? The native host app, the realtime session + coordinator, the SDK/routes, and the Web Shell controls each seem like they could be reviewed independently. A 21K-line PR is very hard to review carefully, and splitting would make each piece easier to reason about and revert.
- CI. The
Test (ubuntu-latest, Node 22.x)check is failing on yamllint —packages/desktop/apps/live-host/electron-builder.ymlhas unquoted strings. Straightforward fix, but it needs to go green. - Direction question. Live Voice is a significant new product surface. Is this something the team wants to take on as a first-party feature, or would it be better suited as an extension / community integration? That's a maintainer call, not something the gate can decide.
Escalating to the maintainer for a direction review. No judgement on the code quality yet — this is about whether the scope and direction align with where the project wants to go before investing in a full review pass.
中文说明
@LaZzyMan 感谢提交——设计文档、测试计划和 fail-closed 策略都考虑得很周全，PR 描述也非常详尽。
不过，这是一个 21,400 行、111 个文件的大型功能，横跨五个 package（core、cli、desktop、sdk-typescript、acp-bridge），包含一个全新的 Electron + Swift 原生 Host、实时语音 provider，以及系统级 macOS 权限（辅助功能、屏幕录制、输入监控）。它还复用了 DashScope 凭证来访问 realtime 模型，并引入了新的 Computer Use 信任边界（Appshot 门禁、屏幕恶意指令处理）。这些恰好是——认证、凭证流转和 Computer Use 信任——需要维护者先对方向表态，然后才能进行如此大规模的代码审查。
几个值得在推进前讨论的问题：
- 范围。 能否拆成一系列更小的 PR？原生 Host 应用、实时会话 + Coordinator、SDK/路由、Web Shell 控件，看起来都可以独立审查。21K 行的 PR 很难仔细审查，拆分后每个部分都更容易推理和回滚。
- CI。
Test (ubuntu-latest, Node 22.x)检查因 yamllint 失败——packages/desktop/apps/live-host/electron-builder.yml中的字符串未使用单引号。修复很简单，但需要先变绿。 - 方向问题。 实时语音是一个重要的新产品面。这是团队希望作为第一方功能承接的，还是更适合作为扩展/社区集成？这是维护者的决定，不是 gate 能判断的。
转交维护者进行方向审查。目前尚未对代码质量做评判——这是关于范围和方向是否与项目目标一致，然后再投入完整的审查。
— Qwen Code · qwen3.8-max-preview
Review:
|
|
Qwen Code review did not complete successfully. Run review failed. See workflow logs for details. A transient error is retried automatically; if you are seeing this, retry with |
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. 2 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here. Not reviewed: chunks 18-20 — briefs not reached (test-only: session-coordinator.test.ts). Not reviewed: chunks 23-24 — briefs not reached (test-only: worker-workspace, provider-credentials tests). Not reviewed: chunks 29-31 — briefs not reached (test-only: run-serve-live, multi-workspace tests). Not reviewed: chunks 35-37 — briefs not reached (test-only: server.test.ts). Not reviewed: chunks 43-45 — briefs not reached (test-only: live-host tests). Not reviewed: 55 of the diff's 61 sections — its prompt was built, but no agent on record was launched with it. Not reviewed: verification and reverse audit — neither the verifier nor the reverse auditor was launched with a prompt this skill builds — the posted findings were ruled on, and the misses the rest of the review left were hunted, if at all, without the briefs this skill certifies against. [Critical] H1 — /live/status and every state broadcast do a synchronous, uncached loadSettings(), including once per output-audio delta. Still stands — not addressed by this diff. [Critical] H2 — create_sub_session with completion: 'sent' now notifies the parent for every caller, not just Live. deliverSentCompletion is called unconditionally. Still stands — not addressed by this diff. [Critical] H3 — The entire live-host test suite never runs in CI. Still stands — not addressed by this diff. [Critical] M1 — The daemon bearer token is written to disk even when Live Voice is disabled. Still stands — not addressed by this diff. [Critical] M2 — The wire protocol and CUA driver pin are duplicated across the desktop isolation boundary with no drift check. Still stands — not addressed by this diff. [Critical] M3 — awaitSentCompletionAcceptance retries at a fixed 100 ms for up to 30 minutes, no backoff. Still stands — not addressed by this diff.
中文说明
已审查。 建议见行内评论。 2 条建议级发现无法锚定到改动行，已丢弃；此处无需进一步处理。 未审查：chunks 18-20 — briefs not reached (test-only: session-coordinator.test.ts)。 未审查：chunks 23-24 — briefs not reached (test-only: worker-workspace, provider-credentials tests)。 未审查：chunks 29-31 — briefs not reached (test-only: run-serve-live, multi-workspace tests)。 未审查：chunks 35-37 — briefs not reached (test-only: server.test.ts)。 未审查：chunks 43-45 — briefs not reached (test-only: live-host tests)。 未审查：diff 61 个片段中的 55 个——它的 prompt 已构建，但没有任何 agent 有记录用它启动过。 未审查：验证与反向审计——验证 agent 与反向审计 agent 都没有用本 skill 构建的 prompt 启动——发布的发现即便被裁定过、评审其余部分遗漏的问题即便被搜寻过，也都缺失了本 skill 用以认证的 brief。 [Critical] H1 — /live/status and every state broadcast do a synchronous, uncached loadSettings(), including once per output-audio delta. Still stands — not addressed by this diff. [Critical] H2 — create_sub_session with completion: 'sent' now notifies the parent for every caller, not just Live. deliverSentCompletion is called unconditionally. Still stands — not addressed by this diff. [Critical] H3 — The entire live-host test suite never runs in CI. Still stands — not addressed by this diff. [Critical] M1 — The daemon bearer token is written to disk even when Live Voice is disabled. Still stands — not addressed by this diff. [Critical] M2 — The wire protocol and CUA driver pin are duplicated across the desktop isolation boundary with no drift check. Still stands — not addressed by this diff. [Critical] M3 — awaitSentCompletionAcceptance retries at a fixed 100 ms for up to 30 minutes, no backoff. Still stands — not addressed by this diff.
— qwen3.7-max via Qwen Code /review
- Quote all strings in electron-builder.yml to fix yamllint CI failure - Gate discovery publish on liveVoiceEnabledAtBoot to avoid writing bearer token to disk when Live Voice is disabled (M1) - Add child identity guard to CommandMonitor stdout/stderr handlers to prevent stale helper output from corrupting the new buffer (M4) - Add exponential backoff to sent-completion delivery retry (M3) - Skip broadcastState when setCallState/setTranscript value is unchanged to reduce per-audio-delta overhead (H1) - Document sent-mode completion notification in module docstring (H2) - Remove dead protocol/nonce aliases from readDiscoveryFile - Fix single instance lock fall-through with process.exit(0)
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/100 轮）。改动内容与我反驳保留之处如下： Review feedback addressedSame-run verification repairThe previous commit (
Automated reviewer (CHANGES_REQUESTED)
@wenshao's reviewHigh
Medium
Low / nits
Verification
中文说明已处理的审查反馈同轮验证修复上一个提交（
自动审查器（CHANGES_REQUESTED）
@wenshao 的审查High
Medium
Low / nits
验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审；第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
中文说明
— qwen3.7-max via Qwen Code /review
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
中文说明
— qwen3.7-max via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 2/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 2/100 轮）。改动内容与我反驳保留之处如下： Review Feedback — Round SummaryCritical findings
Suggestions
Review-body Critical findings (H1–H3, M1–M3)These six findings were posted in the review body (not as inline comments) and are marked "Still stands — not addressed by this diff." They describe pre-existing architectural concerns in the PR (uncached Verification
中文说明审查反馈 — 本轮总结Critical 发现
建议
审查正文中的 Critical 发现（H1–H3、M1–M3）这六项发现发布在审查正文中（非行内评论），标记为"仍然存在——本次 diff 未解决"。它们描述了 PR 中预先存在的架构问题（未缓存的 验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审；第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
|
Addressed the confirmed round-5 findings in e84799c and synced the branch with current main in 4e7761c.
Validation: ACP bridge 479 tests, affected CLI suites 508 tests, Live Host 67 tests, and all three affected typechecks pass. Local arm64 and x64 packages were inspected with electron-fuses and show the intended fuse states; the release find expression matches 2 top-level apps instead of 10 paths including helpers. No install, app launch, published release, or takeover automation was run. |
|
@qwen-code /triage |
|
Sandboxed verification: The verification ran, but its result artifact could not be retrieved for publishing, so there is nothing to report here. The run log still has the agent output; re-run 中文 — 判定：
|
|
本地合并前自查已完成并收敛：
|
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
yiliang114
left a comment
yiliang114
left a comment
There was a problem hiding this comment.
LGTM, no blockers — unusually hardening-conscious for a feature this size: release workflow permissions/injection clean (publish gated to workflow_dispatch+main+repo check, SemVer-validated, SHA-pinned actions, installer trust ends at Developer ID+notarization not just GitHub), host-daemon auth nonce+timing-safe on loopback, audio bounded with backpressure, live-session isolation unforgeable, raw audio never persisted by default. Non-blocking P2s to address or document before/after merge: (1) experimental.liveVoice.apiKey stored plaintext in settings.json — move to OS keychain or warn loudly in docs; (2) Live tools (incl. capture_screen_context) bypass the permission pipeline with finalPermission allow — users can't deny screen capture per-call inside a Live session; add an on-screen capture indicator and document that permission rules don't apply; (3) create_thread/send_message_to_thread queue agent work in any registered workspace from voice alone — release-note callout. P3s: add stapler validate in inspectApp for CI parity; document QWEN_LIVE_DIAGNOSTICS PCM dumps.
|
Released in v0.21.6. |
6 participants
What this PR does
This PR adds experimental Live Voice onboarding and the Codex-parity Live architecture to WebShell on macOS. Live Voice is disabled by default and is not exposed by the CLI/TUI, API-only daemons, non-macOS daemons, or WebShell instances without the native integration surface.
Users configure a dedicated DashScope key for
qwen3.5-omni-plus-realtimeand a global shortcut in the WebShell Experimental settings. Enabling the feature requires confirmation, validates the realtime credential, automatically downloads and verifies the correct signed Qwen Live Host build, installs it in/Applications, launches it, and guides the user through Microphone, Accessibility, and Screen Recording permissions. The default shortcut is Command+E, and the existing one-shot dictation button remains available beside Live Voice.Each call attaches a Qwen Omni Realtime conversation to one persistent projectless Qwen Code session. Realtime answers self-contained dialogue directly and hands off requests that need tools or deeper execution to the same ordinary session, which retains its normal tools and permission pipeline while adding Live Appshot and task list/read/wait/send/create tools. Backend agent messages return incrementally to the active realtime conversation, and new speech can interrupt or steer the same turn.
Appshot is a built-in native Qwen Live Host capability. It does not use an external MCP, CUA driver, helper application, or process launcher. A screen request is handed to the backend session, which invokes the Live-only screen-context tool over the authenticated Host channel.
Desktop releases now build arm64 and x64 Qwen Live Host ZIP/DMG assets plus a versioned manifest containing protocol, bundle, size, and SHA-256 data. Published builds require the expected Developer ID team, Gatekeeper acceptance, notarization, and stapling; stable assets are uploaded to the fixed desktop release feed consumed by onboarding.
Why it's needed
WebShell already supports one-shot dictation, but it did not provide an always-available full-duplex coding conversation that can be invoked from any macOS app, remain attached to durable Qwen Code work, inspect the visible app on request, and follow existing or newly created tasks. This change supplies that experience while keeping model authority, tool permissions, installation, credentials, and Appshot ownership explicit and fail-closed.
Reviewer Test Plan
How to verify
Evidence (Before & After)
Before: WebShell exposed only browser dictation and had no supported global Live onboarding, projectless Live group, realtime-owned dialogue, task follow-up surface, built-in Appshot channel, or release-delivered native Host.
After: The macOS WebShell Experimental panel provides the complete disabled-by-default key, shortcut, confirmation, installation, launch, and permission flow. The Live overlay, realtime conversation, selective same-session handoff, incremental backend return, interruption, task tools, existing permission UI, internal Appshot, projectless session links, and independent dictation control have been exercised in real user scenarios. Static architecture checks confirm no external CUA/MCP dependency or Input Monitoring path remains.
Final validation passed full-repository lint, build, and typecheck; targeted setup, WebShell onboarding, Live lifecycle, SDK, bridge, and Host suites; dual-architecture release-manifest SHA/size verification; workflow lint; and two clean source/packaging audits. The production signed-and-notarized auto-install path remains gated on an actual release run with repository signing and notarization secrets.
Tested on
Environment (optional)
macOS, Node.js 22, Qwen Code WebShell, Qwen Live Host arm64/x64 local packages, and the live DashScope
qwen3.5-omni-plus-realtimeservice.Risk & Scope
Linked Issues
N/A
中文说明
本 PR 做了什么
本 PR 为 macOS WebShell 增加实验性 Live Voice 上手流程和对齐 Codex 的 Live 架构。Live Voice 默认关闭，并且不会在 CLI/TUI、仅 API daemon、非 macOS daemon 或没有原生集成面的 WebShell 中暴露。
用户在 WebShell 的实验性设置中配置
qwen3.5-omni-plus-realtime专用 DashScope key 和全局快捷键。开启功能时必须确认，系统会验证 realtime 凭证，自动下载并校验正确架构的已签名 Qwen Live Host，安装到/Applications、启动应用，并引导完成麦克风、辅助功能和屏幕录制三项授权。默认快捷键为 Command+E，原有的一次性语音输入按钮继续与 Live Voice 并存。每次通话都会把一段 Qwen Omni Realtime 对话附加到一个持久的无项目 Qwen Code session。Realtime 直接回答自包含对话；需要工具或更深执行时，handoff 到同一个普通 session。该 session 保留正常工具和权限流程，同时增加 Live Appshot 以及任务 list/read/wait/send/create 工具。后台 Agent 消息会增量返回仍处于活动状态的 realtime 对话，新语音可以打断或引导同一个 turn。
Appshot 是 Qwen Live Host 内置的原生能力，不依赖外置 MCP、CUA driver、辅助应用或进程启动器。屏幕请求先 handoff 到后台 session，再通过经过认证的 Host 通道调用 Live 专用屏幕上下文工具。
Desktop release 现在会构建 arm64 和 x64 两种 Qwen Live Host ZIP/DMG 产物，并生成包含协议、bundle、大小和 SHA-256 的版本化 manifest。正式发布必须通过指定 Developer ID team、Gatekeeper、公证和 stapling 校验；稳定产物会上传到 onboarding 使用的固定 desktop release feed。
为什么需要
WebShell 已经支持一次性语音输入，但此前无法提供可从任意 macOS 应用唤起、保持全双工语音交互、附着于持久 Qwen Code 工作、按需查看当前应用并跟进已有或新建任务的能力。本改动补齐该体验，同时让模型权威、工具权限、安装、凭证和 Appshot 归属保持明确并 fail-closed。
Reviewer 测试计划
如何验证
证据（Before & After）
Before：WebShell 仅暴露浏览器语音输入，没有受支持的全局 Live onboarding、无项目 Live 分组、Realtime 对话权威、任务跟进面、内置 Appshot 通道或随 release 交付的原生 Host。
After：macOS WebShell 实验性面板提供完整的默认关闭、key、快捷键、确认、安装、启动和权限引导流程。Live 悬浮 UI、Realtime 对话、选择性同 session handoff、后台增量返回、打断、任务工具、现有权限 UI、内部 Appshot、无项目 session 链接和独立语音输入控件均已在真实用户场景中验证；静态架构审计确认不再存在外部 CUA/MCP 依赖或输入监控路径。
最终验证通过全仓 lint、build 和 typecheck，setup、WebShell onboarding、Live 生命周期、SDK、bridge 和 Host 的针对性测试，双架构 release manifest 的 SHA/大小校验，workflow lint，以及两轮干净的源码和打包架构审计。正式签名及公证后的自动安装仍需在具备仓库签名和公证 secrets 的真实 release 中完成。
测试平台
环境（可选）
macOS、Node.js 22、Qwen Code WebShell、Qwen Live Host arm64/x64 本地产物，以及线上 DashScope
qwen3.5-omni-plus-realtime服务。风险与范围
关联 Issue
N/A