[ANT1OFF]

Adds an Accurate voice-input mode. Nemotron still streams the live words at the caret. At each 1.2 s pause, and on stop, the speech process trims the utterance's audio with Silero VAD v5 and runs Parakeet TDT 0.6B v2 (int8, sherpa-onnx OfflineRecognizer) over it. It then commits that text, with punctuation and capitals, in place of the streamed words. Live is the old behaviour. Accurate is the default.

Measurements

Apple M5 Pro (18 cores, 24 GB), sherpa-onnx 2.0.0 CPU, 2 threads, run in Node 26 through the same addon the app loads. Other checkouts were running on this machine (load average 10–40), so the timings are pessimistic.

The study's WAVs aren't in the tree, so I used two corpora:

• Real read speech: hf-internal-testing/librispeech_asr_dummy, 73 clips, 481 s.
• Synthetic: macOS say, 20 developer-prompt sentences × 4 voices, 375 s.

Neither is the user's voice or microphone. WER is case- and punctuation-insensitive.

	Nemotron 560 ms (today)	Parakeet v2 int8	Parakeet v3 int8
WER, LibriSpeech (1150 words)	6.43 %	3.04 %	4.17 %
WER, say (1216 words)	0.49 %	0.41 %	0.49 %
Decode, 4–6 s clip, p50 / p95 (load ~10)	(streams)	240 / 307 ms	277 / 388 ms
Decode, 4–6 s clip, p50 / p95 (load ~40)	(streams)	613 / 1520 ms	765 / 3964 ms
RTF over the corpus (load ~10)	0.14–0.22	0.05	0.057
Load	0.9–1.7 s	1.6–3.4 s (5.5 s at load 40)	2.3–3.7 s
Peak memory, alone	2.2–2.3 GB	1.9–2.3 GB	2.0–2.1 GB

• Both models resident in one process: 3.0–3.1 GB steady footprint, 3.38 GB peak; Nemotron alone is ~1.7 GB steady.

• Threads for the pass: 1 thread p50 734 ms; 2 threads 203–240 ms; 4 threads 218 ms; 6 threads 310 ms. So it uses 2.

• CoreML (provider: "coreml"): p50 2016 ms, p95 4749 ms, 12.7 s load, 3.3 GB peak, same WER. Rejected.

• Silero VAD v5, real-time factor 0.008. On LibriSpeech clips with 0.5 s / 1.2 s of −60 dBFS noise around them:

  Speech level	WER untrimmed	WER trimmed	Clips missed
  Normal	3.74 %	3.04 %	0
  20 dB quieter	4.26 %	3.30 %	0
  30 dB quieter	5.83 %	6.17 %	0

  Digital silence and white noise at −60…−20 dBFS gave no text from either model, so the "thank you" hallucination did not reproduce with stationary noise. The VAD skip guards against it anyway.

Design

• Speech process: service.ts keeps each utterance's audio, including the 300 ms lead and the stop tail. On an endpoint or stop it:
  • runs planClip: keeps 250 ms around the VAD's speech, and returns nothing when no speech was found;
  • transcribes the clip with decodeAsync, off the main thread, so a cancel still lands;
  • commits the result.
• Bounds:
  • Clips are cut at 30 s. When speech runs into the cut, the clip ends at the last pause and the rest carries into the next utterance.
  • A pass that takes over 4 s (load included) commits the streamed text instead. That is short of the page's 5 s stop wait.
  • Frames queue while an utterance commits, so the next partials never overtake its final.
• Loading: Parakeet and Silero load on the first Accurate prewarm or session, and are freed with the process on the existing 5-minute idle timer, or by release.
• Fallback: an Accurate session whose models aren't installed runs Live; main decides this from verified.json, never from the page.
• Logs: an accurate pass line carries audioMs, ms and fellBack only, never text. A test asserts this.
• Contracts (additive):
  • SpeechOpenRequest.mode, and a mode argument on DesktopSpeech.prewarm;
  • voice.mode in settings (speechModeOf, Accurate by default);
  • SpeechStatus.totalMemoryBytes;
  • SpeechRecognizer is now a union with sherpa-onnx-offline-transducer and sherpa-onnx-silero-vad;
  • an optional SpeechModelSource.url.
• No new port message: final already replaces the partial; its doc now says the text can differ from the last partial.

Why Accurate is the default

Parakeet halves the word errors on real speech (6.43 → 3.04 %) and adds punctuation. On an idle machine it costs about a quarter of a second after each pause. Accurate is offered only with consent, and it degrades to Live when its models are missing, too slow or broken. Machines under 16 GB see only the Live download on the first-run consent screen, because both models together take about 3 GB.

Models and consent

• Manifest: parakeet-tdt-0.6b-v2-en-int8 (csukuangfj @ 1ab9323…, 661 MB, from nvidia/parakeet-tdt-0.6b-v2 @ ae9ad07…) and silero-vad-v5 (csukuangfj/vad @ fba88cd…, 2.3 MB). Each file has a pinned size and the sha256 that Hugging Face's LFS serves; the Silero file's sha256 also equals upstream files/silero_vad.onnx at tag v5.0.
• Consent: download, resume, verification and consent use the existing store.
  • The first-run screen lists all three models, each with its own licence link and attribution: NVIDIA Open Model License, CC BY 4.0 (author, licence, and "converted to int8 ONNX"), and MIT. It offers "Download 1.3 GB" or "Live only, 662 MB".
  • Adding Accurate later has its own consent screen.
  • Turning voice off deletes all of it.
• Notices: packages/shell/src/modules/speech/MODEL-NOTICES.md (with the full Silero MIT text) is now a kept notice in tools/notices. A test fails if any pinned model's attribution or licence URL is missing from it.

Screenshots (mock shell)

Before	After

Checks

• bun run typecheck, bun run lint, bun run fmt:check, bun run check:scaffold: pass.
• vitest over the speech process, the speech module, contracts, web settings/dictation and tools/notices: 1024 tests pass.
• apps/web/e2e/dictation.spec.ts: 13 pass, including a new Accurate consent/download/mode test.
• Live, real runtime (GRAM_LIVE_SPEECH=1, with GRAM_LIVE_SPEECH_ACCURATE and GRAM_LIVE_SPEECH_VAD): both live tests pass on the Nemotron export's test_wavs/0.wav followed by 3 s of noise. The first final is Parakeet's punctuated "yellow lamps" sentence; the noise commits nothing.
• I didn't run my own full bun run check.

Known limits / not verified

• Not tested with a microphone or a real voice, and not in the packaged Electron utility process. The live test runs the same addon in Node. Vad.front(false) is used because Electron refuses external buffers; that is untested inside Electron.
• WER is on read LibriSpeech and TTS clips, not dictated developer prompts.
• English only. Parakeet v3 (25 European languages) is a drop-in manifest entry, but Nemotron's live partials are English-only, so a language choice is a follow-up.
• Linux: the code is platform-neutral (Node-API OfflineRecognizer and Vad, CPU provider, no CoreML) and the Linux voice gate is untouched. The path is ready once a TTS-free sherpa-onnx-linux-x64 build exists; the real-time factor on Linux x64 is unmeasured.
• Rebased onto feat(voice): start capture at the press, and an opt-in ready microphone #81 (Keep microphone ready): the Settings rows sit side by side, and use-dictation.ts passes mode through to start and prewarm. After the rebase: typecheck, lint, fmt, scaffold and 1039 vitest pass; e2e/dictation.spec.ts 15/15.