Skip to content

feat(web): add verified Android STT bridge - #95191

Draft
kvnloo wants to merge 7 commits into
NousResearch:mainfrom
kvnloo:feat/android-stt-webui-bridge
Draft

kvnloo wants to merge 7 commits into
NousResearch:mainfrom
kvnloo:feat/android-stt-webui-bridge

Conversation

@kvnloo

@kvnloo kvnloo commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an optional, typed Android WebView STT bridge to the dashboard voice control
  • require availability and readiness handshakes before accepting native transcripts
  • keep partial transcripts display-only and submit each accepted final exactly once
  • reject duplicate, stale-generation, and stale-bridge callbacks, including retained callbacks after browser fallback
  • preserve Chrome Web Speech as the fallback and leave assistant-final TTS unchanged

Stack dependency

This Android follow-up now inherits the independently approved and published browser WebUI voice base at exact commit 84b4cdece0e90f627fca56ef92122a92c0d8f4c0 from #95193. The Android delta is the patch-equivalent replay of the original two Android commits (ee1357982, 3144ceeac) on that base, followed by the focused C4/C6 repair. Review the Android delta as 84b4cdece..91b5872130; do not merge this PR before #95193 lands and this branch is rebased onto main.

Android companion contract

The concrete Android companion candidate is commit 836dd8cee870f020d3db0ce355846f9f947fe242. It installs the zer0Voice AndroidX WebView listener with native exact-origin enforcement. The JavaScript payload remains origin-neutral and contains only protocol version, command/event names, availability mode, and transcript text; it adds no credentials or origin/auth bypass.

The v1 flow requires checkAvailability -> availability -> start -> ready. Native unavailable, error, timeout, and terminal-without-final paths fall back to browser Web Speech. Partials never submit. Finals submit through the existing exactly-once PTY path.

C3-C6 closure

Exact approved head: 91b5872130ace7d0946045cdcded8f6a3af5e2fe.

  • C3 inherited from feat(web): add mobile browser voice conversation #95193: synchronized PTY Return waits for the matching xterm write callback, remains bound to the captured live socket, and visibly fails if that socket closes or is replaced.
  • C4: browser recognition language follows navigator.language rather than a hardcoded locale.
  • C5 inherited from feat(web): add mobile browser voice conversation #95193: speakAssistantFinal internally guards unavailable SpeechSynthesisUtterance and returns without synthesis.
  • C6: syntactically malformed native bridge payloads warn and are ignored; they do not silently trigger browser fallback, so the selected native handshake can continue.

Verification

Independent review approved exact head 91b5872130ace7d0946045cdcded8f6a3af5e2fe.

  • focused Vitest: 4 files / 34 tests passed
  • full WebUI Vitest: 39 files / 306 tests passed
  • TypeScript typecheck passed
  • ESLint: 0 errors / 26 pre-existing warnings
  • production build passed
  • git diff --check passed
  • RED/GREEN, sabotage assessment, and three simplification passes completed

Evidence boundary

No real-device or on-device-success claim is made by this PR. Physical Galaxy S25 verification remains required before publication of any such claim; this PR currently proves the WebUI contract and fallback behavior only.

Part of #94462.

Inherited reduced-motion repair

This branch is now rebuilt on approved browser base 65539bc3361faf7afca31872db38eae70754052b; exact head f843f20d9ddb7d1ac7f74e6d6fb754413cb58c76 preserves the three Android-specific commits patch-equivalently with no duplicate browser fixes. The inherited listening indicator respects reduced-motion preference.

Verification: focused voice/PTTY/ChatPage 4 files / 35 tests passed; full WebUI 39 files / 307 tests passed; typecheck, production build, changed-area ESLint and git diff --check passed; synthetic current-main merge remained conflict-free. Independent review approved both repaired stack revisions.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/dashboard Web dashboard / control panel UI (dashboard/, landing) tool/tts Text-to-speech and transcription labels Aug 26, 2026
@kvnloo

kvnloo commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Voice stack (from #94462): merge order — #95193 browser base → #95191 Android STT → #95180 desktop lifetimes → #95147 realtime provider seam. Review/merge in that order; later PRs depend on earlier ones landing first.

@kvnloo

kvnloo commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Polish follow-up (author self-review): a few body sections are truncated mid-sentence and should be completed or trimmed before merge review:

  • verification bullet ends mid-line ("- focus") — finish or remove the dangling fragment.
    @kvnloo to tidy when convenient; no code change needed.

@kvnloo
kvnloo marked this pull request as draft August 26, 2026 04:26
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

This PR adds a well-architected Android STT bridge for the web UI, with thoughtful fallback handling between a native bridge (window.zer0Voice) and browser SpeechRecognition. The generation-based stale-callback rejection pattern is consistently applied across both native and browser paths, and the test suite (308 lines) covers an impressive range of edge cases: restart backoff, fatal permission errors, native handshake timeout, stale native callbacks after fallback, unmount cleanup, and the Android-specific speech-endedfinal sequence.

A few concerns:

  1. submitVoiceTranscriptToPty partial-send vulnerability (pty-voice-submit.ts:22-26): The transcript is sent in one frame and \r in a separate frame 100ms later. If the WebSocket closes between the two sends, the transcript text sits in the composer without being submitted, and there's no recovery mechanism — the user sees their text but nothing happens. Consider either: (a) sending both in one frame with a different delimiter strategy, or (b) surfacing a user-visible error when the Return send fails so the user knows to manually submit.

  2. SpeechRecognition.lang hardcoded to "en-US" (ChatVoiceControl.tsx:168): The language is fixed to US English. For a voice input feature on Android (which has diverse locale support), this limits usefulness for non-English users. Consider reading from navigator.language or making it configurable.

  3. speakAssistantFinal doesn't check speechSynthesis availability before construction (assistant-voice-output.ts:25): The caller in ChatPage.tsx guards with "speechSynthesis" in window, but the speakAssistantFinal function itself constructs new SpeechSynthesisUtterance(text) unconditionally. If called in an environment without SpeechSynthesis (e.g., an older browser), this would throw. Adding a guard inside the function would make it more robust as a library utility.

  4. Native bridge onmessage handler does a JSON.parse on every message (ChatVoiceControl.tsx:210): If the native bridge sends non-JSON data (e.g., a debug log), the catch block triggers fallbackToBrowser. This is aggressive — a single malformed message causes an immediate fallback to browser speech, even if the native bridge was functioning correctly. Consider logging and ignoring non-JSON messages instead of falling back.

The review artifact (docs/review-artifacts/) with SHA-256-verified patch provenance is unusual but provides strong auditability. The truth-boundary section explicitly disclaiming unverified claims (no on-device/offline recognition, no real-device proof) is commendable scientific honesty.

@kvnloo
kvnloo force-pushed the feat/android-stt-webui-bridge branch from 3144cee to 91b5872 Compare August 28, 2026 21:52
@kvnloo

kvnloo commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Closed C3-C6 at exact approved head 91b5872130ace7d0946045cdcded8f6a3af5e2fe: inherited synchronized PTY Return and the internal synthesis guard from #95193, switched recognition locale to navigator.language, and made malformed native payloads warn-and-ignore without browser fallback. RED→GREEN and sabotage checks pin locale, malformed-payload logging/native continuation, PTY callback ordering/socket failure, and synthesis capability failure. Verification: focused 4 files/34 tests; full WebUI 39 files/306 tests; typecheck; lint 0 errors/26 pre-existing warnings; production build; diff-check; three simplification passes.

@kvnloo

kvnloo commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Comment assumes head f843f20d9ddb7d1ac7f74e6d6fb754413cb58c76; void if moved.

KEEP — Android STT bridge consumer only (after #95193)

CHECK

One-click close / ordering

Landing-evidence

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P3 Low — cosmetic, nice to have tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants