fix(stt): kill faster-whisper silence hallucinations — VAD, no conditioning, confidence gate

[teknium1]

fix(stt): kill faster-whisper silence hallucinations at the source

Summary

Local faster-whisper STT hallucinated random words — often in other languages — from silence and background noise (reported by Teknium as unacceptable). Root cause: _transcribe_local in tools/transcription_tools.py called model.transcribe with bare {"beam_size": 5} — no VAD, no confidence filtering, and condition_on_previous_text left default-on (a hallucination amplifier: one junk token seeds a run of them).

This is a class fix at the source, not a blocklist extension. Three layers, all owned by one shared kwargs helper (build_local_transcribe_kwargs) that every local-whisper call site goes through:

Layer	Before (bare kwargs)	After (hardened)
VAD	none — silence fed straight to whisper	vad_filter=True (Silero VAD, bundled with faster-whisper 1.2.1) + min_silence_duration_ms: 500. Default on; stt.local.vad: false restores raw behavior for music/ambient transcription
Conditioning	condition_on_previous_text default True	False — one hallucinated token can't seed a self-reinforcing run; negligible cost for voice-note-length audio
Confidence gate	every decoded segment kept	segment dropped only when no_speech_prob > 0.6 AND avg_logprob < -1.0 (openai-whisper's own heuristic shape — the AND means quiet-but-real and mumbled-but-real speech both survive)

The WHISPER_HALLUCINATIONS blocklist in tools/voice_mode.py is deliberately untouched — it remains last-resort defense and should now almost never fire. The stt.language: "en" default from #73100 already reduced cross-language hallucinations; this completes the fix.

E2E evidence (real faster-whisper base, CPU int8, real WAVs)

Test audio generated with ffmpeg (anullsrc silence, anoisesrc pink/white noise) and edge-tts speech, run through the actual _transcribe_local:

Input	Before (main)	After (this branch)
5s pure silence	"You" (hallucinated; segment no_speech_prob=0.705, avg_logprob=-0.793)	"" ✅
5s pink noise (amp 0.05)	""	"" ✅
5s white noise (amp 0.3)	""	"" ✅
edge-tts speech ("Hello world, this is a test of the transcription system.")	"Hello World, this is a test of the transcription system."	"Hello World, this is a test of the transcription system." ✅ unchanged

Also verified with stt.language: "" (auto-detect — the worst case for cross-language junk): silence/noise → empty, speech unchanged.

Config keys (all under stt.local, wired into DEFAULT_CONFIG + cli-config.yaml.example + docs EN/zh-Hans)

stt:
  local:
    vad: true                      # Silero VAD filter — silence never reaches whisper. false = old raw behavior
    vad_min_silence_ms: 500        # min silence (ms) that splits speech chunks when vad is on
    no_speech_prob_threshold: 0.6  # drop a segment only if no_speech_prob > this...
    logprob_threshold: -1.0        # ...AND avg_logprob < this (both must hit — quiet real speech survives)

Call-site audit

_transcribe_local is the only in-repo faster-whisper invocation — the TUI gateway (tui_gateway/server.py) and dashboard (hermes_cli/web_server.py /api/audio/transcribe) both route through transcribe_recording → transcribe_audio → _transcribe_local, so every surface (gateway voice notes, CLI voice mode, desktop, dashboard, wake-word loop) inherits the hardening from the one helper. Any future local-whisper call site must use build_local_transcribe_kwargs instead of hand-rolling kwargs.

Dupe sweep

Searched open PRs/issues for vad, vad_filter, hallucination, no_speech, silence whisper, condition_on_previous_text, faster-whisper. No existing PR implements VAD filtering or confidence gating for the local STT transcription path — nothing to salvage:

• fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083 — rolling-window VAD for barge-in during TTS (recording-side, different subsystem)
• feat(cli): add /voice vad command #19984 / [Feature] /voice vad — hands-free VAD mode via slash command #19927 — /voice vad hands-free recording mode (capture-side VAD, not transcription filtering)
• feat(discord): production-grade voice channel with configurable VAD and auto-join #62499 — Discord voice-channel VAD (platform capture pipeline)
• feat(stt): add hotwords support for STT vocabulary biasing #52726 — hotwords vocabulary biasing (adjacent, deliberately not scope-crept here)

Tests

19 new unit tests in tests/tools/test_stt_silence_hallucinations.py:

• kwargs contract: VAD on by default, off via stt.local.vad: false, conditioning always off, beam_size kept, vad_min_silence_ms configurable with garbage fallback, YAML local: null safe
• confidence gate: high-no_speech + low-logprob dropped; quiet-but-confident speech survives; low-confidence-only (mumbled real speech) survives; missing/plugin segment attrs never dropped; thresholds configurable with garbage fallback
• _transcribe_local wiring: hardened kwargs actually reach the model; off-switch reaches the model; hallucinated segments filtered from the transcript

Sabotage-verified: flipping the VAD default to False fails 5 tests. Full STT suites green (203 passed: test_transcription_tools, test_stt_silence_hallucinations, test_stt_default_language, test_stt_language_resolution, test_managed_media_gateways). Ruff clean.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on bafc581}

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

_{inline evidence upload failed.}

Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso)