Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/design/web-shell-voice-mode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Web Shell voice mode

## Goal

Make the Web Shell voice control honor the workspace voice mode already returned by the daemon.

## Behavior

| Mode | Start | Finish | Cancel |
| ------ | -------------------- | --------------------- | ---------------------- |
| `hold` | Primary pointer down | Matching pointer up | Pointer cancel |
| `tap` | Click while idle | Click while recording | Click while connecting |

The control captures the active pointer so releasing outside the button still finishes a hold. Keyboard-generated clicks retain the existing toggle behavior in either mode.

Once microphone access is ready, audio produced while the socket connects is buffered locally. The socket receives `start`, the buffered audio, and then a deferred `stop`, so releasing during connection does not discard captured speech.

The mode is loaded with the existing workspace voice status and refreshed through the existing settings-version signal. A status request remains fail-closed: the control stays hidden until the current workspace reports that voice is enabled.

## Scope

This change only affects Web Shell input interaction. It does not change the daemon settings contract, transcription transport, model selection, or TUI voice behavior.
159 changes: 155 additions & 4 deletions packages/web-shell/client/voice/VoiceButton.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,16 @@ vi.mock('./useVoiceCapture', () => ({

const mounted: Array<{ root: Root; container: HTMLElement }> = [];

function voiceStatus(enabled: boolean, workspaceCwd = '/tmp/workspace') {
function voiceStatus(
enabled: boolean,
workspaceCwd = '/tmp/workspace',
mode: 'hold' | 'tap' = 'hold',
) {
return {
v: 1 as const,
workspaceCwd,
enabled,
mode: 'hold' as const,
mode,
language: 'en',
voiceModel: null,
availableVoiceModels: [],
Expand Down Expand Up @@ -121,9 +125,25 @@ async function render(disabled: boolean): Promise<HTMLButtonElement> {
return button;
}

const click = (button: HTMLButtonElement) => {
const click = (button: HTMLButtonElement, detail = 1) => {
act(() => {
button.dispatchEvent(new MouseEvent('click', { bubbles: true, detail }));
});
};

const pointer = (
button: HTMLButtonElement,
type: 'pointerdown' | 'pointerup' | 'pointercancel',
pointerId = 1,
mouseButton = 0,
) => {
const event = new MouseEvent(type, {
bubbles: true,
button: mouseButton,
});
Object.defineProperty(event, 'pointerId', { value: pointerId });
act(() => {
button.dispatchEvent(new MouseEvent('click', { bubbles: true }));
button.dispatchEvent(event);
});
};

Expand Down Expand Up @@ -584,6 +604,9 @@ describe('VoiceButton', () => {
});

it('lets a disabled composer stop active dictation', async () => {
mocks.workspaceVoice.mockResolvedValue(
voiceStatus(true, '/tmp/workspace', 'tap'),
);
mocks.capture.status = 'recording';
const button = await render(true);

Expand All @@ -594,6 +617,9 @@ describe('VoiceButton', () => {
});

it('lets a disabled composer abort a connecting dictation', async () => {
mocks.workspaceVoice.mockResolvedValue(
voiceStatus(true, '/tmp/workspace', 'tap'),
);
mocks.capture.status = 'connecting';
const button = await render(true);
mocks.capture.abort.mockClear();
Expand All @@ -604,6 +630,131 @@ describe('VoiceButton', () => {
expect(mocks.capture.abort).toHaveBeenCalledOnce();
});

it('holds to start and releases to stop dictation', async () => {
const { root, container } = mount(false);
await flush();
let button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
const heldButton = button;

pointer(button, 'pointerdown');
expect(mocks.capture.start).toHaveBeenCalledOnce();

mocks.capture.status = 'recording';
act(() => {
root.render(
<VoiceButton
disabled={false}
onInsert={() => {}}
target={legacyTarget}
/>,
);
});
button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
// The recording pill reuses the icon button's DOM node, so the pointer
// capture set on pointerdown survives the status change and the release
// still lands on this element.
expect(button).toBe(heldButton);
pointer(button, 'pointerup');
click(button);

expect(mocks.capture.stop).toHaveBeenCalledOnce();
});

it('stops a hold that is released while connecting', async () => {
const { root, container } = mount(false);
await flush();
let button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
pointer(button, 'pointerdown');

mocks.capture.status = 'connecting';
act(() => {
root.render(
<VoiceButton
disabled={false}
onInsert={() => {}}
target={legacyTarget}
/>,
);
});
button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
pointer(button, 'pointerup');

expect(mocks.capture.stop).toHaveBeenCalledOnce();
});

it('aborts a hold when the pointer is cancelled', async () => {
const { root, container } = mount(false);
await flush();
let button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
pointer(button, 'pointerdown');

mocks.capture.status = 'connecting';
act(() => {
root.render(
<VoiceButton
disabled={false}
onInsert={() => {}}
target={legacyTarget}
/>,
);
});
button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
pointer(button, 'pointercancel');

expect(mocks.capture.abort).toHaveBeenCalledOnce();
});

it('keeps click-to-toggle behavior in tap mode', async () => {
mocks.workspaceVoice.mockResolvedValue(
voiceStatus(true, '/tmp/workspace', 'tap'),
);
const { root, container } = mount(false);
await flush();
let button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');

click(button);
expect(mocks.capture.start).toHaveBeenCalledOnce();

mocks.capture.status = 'recording';
act(() => {
root.render(
<VoiceButton
disabled={false}
onInsert={() => {}}
target={legacyTarget}
/>,
);
});
button = container.querySelector('button');
if (!button) throw new Error('VoiceButton did not render');
click(button);

expect(mocks.capture.stop).toHaveBeenCalledOnce();
});

it('ignores mouse click in hold mode', async () => {
const button = await render(false);

click(button); // detail defaults to 1 (real pointer click)

expect(mocks.capture.start).not.toHaveBeenCalled();
});

it('allows keyboard activation in hold mode', async () => {
const button = await render(false);

click(button, 0);

expect(mocks.capture.start).toHaveBeenCalledOnce();
});

it('reports whether voice capture is active', async () => {
const onActiveChange = vi.fn();
const { root } = mount(false, legacyTarget, onActiveChange);
Expand Down
83 changes: 71 additions & 12 deletions packages/web-shell/client/voice/VoiceButton.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import type React from 'react';
import { useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react';
import type { DaemonVoiceMode } from '@qwen-code/sdk';
import {
useConnection,
useWorkspace,
Expand Down Expand Up @@ -110,10 +111,13 @@ export function VoiceButton({
const [voiceGate, setVoiceGate] = useState<{
queryKey: string | undefined;
enabled: boolean;
mode: DaemonVoiceMode;
}>(() => ({
queryKey,
enabled: false,
mode: 'hold',
}));
const holdPointerIdRef = useRef<number | null>(null);
const targetRef = useRef(target);
targetRef.current = target;
const requestGenerationRef = useRef(0);
Expand All @@ -135,6 +139,7 @@ export function VoiceButton({
setVoiceGate({
queryKey,
enabled: false,
mode: 'hold',
});
if (
!requestTarget ||
Expand All @@ -151,6 +156,7 @@ export function VoiceButton({
setVoiceGate({
queryKey,
enabled: voice.enabled === true,
mode: voice.mode,
});
}
})
Expand Down Expand Up @@ -303,6 +309,63 @@ export function VoiceButton({
// composer is disabled (e.g. mid-turn) — only starting a new one is blocked.
const canCancel = isRecording || isConnecting;

const handleClick = (event: React.MouseEvent<HTMLButtonElement>) => {
if (voiceGate.mode === 'hold' && event.detail !== 0) return;
if (isRecording) {
stop();
} else if (isConnecting) {
abort();
} else if (!disabled) {
setNoticeMessage(undefined);
start();
}
};

const handlePointerDown = (event: React.PointerEvent<HTMLButtonElement>) => {
if (
voiceGate.mode !== 'hold' ||
event.button !== 0 ||
disabled ||
isConnecting ||
isRecording
) {
return;
}
event.preventDefault();
holdPointerIdRef.current = event.pointerId;
try {
event.currentTarget.setPointerCapture(event.pointerId);
} catch {
Comment on lines +335 to +338

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined as a false positive — leaving this thread open for a maintainer to confirm.

The finding's load-bearing premise is that when status transitions to recording, React unmounts the icon <button> and mounts a structurally different pill <button>, which (per Pointer Events §5.3) would implicitly release pointer capture. That unmount does not happen here. Both the idle/connecting branch and the isRecording branch render a <button> at the same tree position with no key, so React's reconciler keeps the same DOM node and only patches its className, children, and handlers — implicit capture release is triggered by node removal, which never occurs.

I verified this empirically rather than arguing it from the spec: a DOM-identity check across the idle→recording transition returns the identical node (expect(recordingButton).toBe(idleButton) passes). Because the captured element stays in the DOM, the capture set in handlePointerDown survives, the subsequent pointerup is still delivered to it, and handlePointerUp → stop() fires — including for a release outside the button, so the design guarantee holds. The event.detail !== 0 click guard is irrelevant to this path; the pointer handlers, not the click handler, drive hold mode.

This also means the existing test is faithful to a real browser, not masking the issue: the re-queried button it dispatches pointerup on is the same captured node a real browser would target. To lock the invariant in, I added that one-line DOM-identity assertion (with a short why-comment) to holds to start and releases to stop dictation.

I did not apply the suggested refactor (a single stable button, or a document-level pointerup listener): both would add real complexity to fix a defect that does not exist. Happy to revisit if I've misread React's reconciliation here.

中文说明

拒绝此项,判定为误报 —— 线程保持打开,留待 maintainer 确认。

该结论的关键前提是:当状态切换到 recording 时,React 会卸载图标 <button> 并挂载一个结构不同的 pill <button>,从而(依据 Pointer Events §5.3)隐式释放指针捕获。但这里并不会发生这种卸载。idle/connecting 分支与 isRecording 分支都在同一树位置、且没有 key 渲染一个 <button>,因此 React 的协调器会保留同一个 DOM 节点,只更新它的 className、children 和事件处理函数 —— 隐式释放捕获是由节点被移除触发的,而这里从未发生移除。

我没有仅凭规范推断,而是做了实证验证:在 idle→recording 切换前后做 DOM 同一性检查,返回的是完全相同的节点(expect(recordingButton).toBe(idleButton) 通过)。由于被捕获的元素始终留在 DOM 中,handlePointerDown 中设置的捕获得以保留,随后的 pointerup 仍会派发到该节点,handlePointerUp → stop() 会触发 —— 即使在按钮外释放也成立,因此设计保证成立。event.detail !== 0 的 click 守卫与此路径无关;hold 模式由指针处理函数驱动,而非 click 处理函数。

这也意味着现有测试忠实反映了真实浏览器,并未掩盖问题:测试重新查询到的、并在其上派发 pointerup 的按钮,正是真实浏览器会命中的同一个被捕获节点。为锁定这一不变量,我在 holds to start and releases to stop dictation 中加入了这一行 DOM 同一性断言(以及一条简短的 why 注释)。

我没有采用建议中的重构(单一稳定按钮,或 document 级的 pointerup 监听器):两者都会为一个并不存在的缺陷增加实实在在的复杂度。如果我对 React 协调行为的理解有误,欢迎重新讨论。

// Pointer capture is best-effort; release on the button still works.
}
setNoticeMessage(undefined);
start();
};

const handlePointerUp = (event: React.PointerEvent<HTMLButtonElement>) => {
if (
voiceGate.mode !== 'hold' ||
holdPointerIdRef.current !== event.pointerId
) {
return;
}
holdPointerIdRef.current = null;
if (isConnecting || isRecording) stop();
Comment thread
callmeYe marked this conversation as resolved.
Resolved
};

const handlePointerCancel = (
event: React.PointerEvent<HTMLButtonElement>,
) => {
if (
voiceGate.mode !== 'hold' ||
holdPointerIdRef.current !== event.pointerId
) {
return;
}
holdPointerIdRef.current = null;
if (isConnecting || isRecording) abort();
};

const label = isRecording
? t('voice.stopDictation')
: isTranscribing
Expand All @@ -321,7 +384,10 @@ export function VoiceButton({
<button
type="button"
className={styles.pill}
onClick={() => stop()}
onClick={handleClick}
onPointerDown={handlePointerDown}
onPointerUp={handlePointerUp}
onPointerCancel={handlePointerCancel}
aria-label={label}
title={label}
>
Expand Down Expand Up @@ -365,17 +431,10 @@ export function VoiceButton({
<button
type="button"
className={iconClass}
onClick={() => {
if (isConnecting) {
abort();
} else if (disabled) {
return;
} else {
// idle / error / notice → (re)start
setNoticeMessage(undefined);
start();
}
}}
onClick={handleClick}
onPointerDown={handlePointerDown}
onPointerUp={handlePointerUp}
onPointerCancel={handlePointerCancel}
disabled={Boolean(disabled) && !canCancel}
aria-label={label}
title={errorMessage ?? noticeMessage ?? label}
Expand Down
Loading
Loading