STT v3 — route hermes-agent to whisper.service HTTP + cloud OpenAI primary - #3
Merged
Merged
Conversation
…I primary
Fixes the local STT mismatch where hermes-agent loaded faster-whisper 'base'
in-process while whisper.service was already running with ggml-large-v3-turbo
on port 8765 — the better model was deployed but never used.
tools/transcription_tools.py:
- New _transcribe_via_whisper_service() that POSTs OGG audio to a
configurable HTTP endpoint (whisper.cpp server). Multipart manually
constructed via stdlib urllib so no new pip dep, and works under
SECCOMP without subprocess.
- _transcribe_local() now tries the HTTP endpoint first (when configured
via stt.local.endpoint), falls back to faster-whisper in-process.
tools/voice_io.py:
- Tracked for the first time. Implements the WS16 voice loop client
talking to whisper.service (STT 8765) and tts.service (TTS 8766) via
HTTP. Used by the Discord adapter for STT on inbound vocals and TTS
on outbound native voice messages. Pure stdlib + aiohttp; no
subprocess (compatible with hermes-agent.service SECCOMP).
uv.lock:
- Auto-updated by uv during the session; included for reproducibility.
User-facing config note: ~/.hermes/config.yaml stt.local.endpoint is a new
optional key. When set (e.g. http://127.0.0.1:8765/inference) the
transcription router routes local calls to whisper.service first. When
unset, the legacy faster-whisper in-process path is preserved.
Bench validation (Claude, 2026-05-03 on 4 archived audios):
whisper-1 cloud: 1.2-4.6s, 1/4 hallucinations (disqualified)
gpt-4o-mini-transcribe: 0.7-2.3s, 0/4 hallucinations (recommended)
whisper.service L3T local: 23-47s, 0/4 hallucinations (good fallback)
Smoke test post-deploy: _transcribe_via_whisper_service returns
success=True with correct French transcript via large-v3-turbo. Live
vocal at 18:35 UTC transcribed end-to-end through the new route.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the STT path used by hermes-agent for inbound voice messages from
faster-whisper base(in-process, ~39M params) to:gpt-4o-mini-transcribe(sub-second, $0.003/min, no hallucinations on 4-audio bench)whisper.serviceHTTP athttp://127.0.0.1:8765/inference(whisper.cpp server already running withggml-large-v3-turbo)The local fallback closes a longstanding mismatch:
whisper.servicewas already deployed with the large-v3-turbo model but hermes-agent silently loaded the much smallerbasemodel in-process viafaster-whisper.What's in the diff
tools/transcription_tools.py_transcribe_via_whisper_service(file_path, endpoint, language)function. Pure stdliburllib(norequestsdep), manually-constructed multipart form-data, OpenAI-compatible response shape. Returns same dict as legacy_transcribe_local()._transcribe_local()now tries the HTTP endpoint first (whenstt.local.endpointis configured), falls back to in-processfaster-whisperon any HTTP failure.tools/voice_io.py(newly tracked)uv.lockValidation
Bench performed by Claude on 4 real audios (Khéri's Discord vocals, archived at
~/.hermes/cache/audio/):whisper-1(cloud)gpt-4o-mini-transcribe(cloud)whisper.service large-v3-turbo(local)gpt-4o-mini-transcriberetained as primary.whisper-1disqualified on hallucination rate.whisper.serviceretained as fallback (slow but free + already deployed).User-facing config
Requires
~/.hermes/config.yamlto be updated to set:Config not in this PR (it's user runtime, not in repo). Applied to prod separately.
Test plan
_transcribe_via_whisper_serviceagainst archived OGG → success + correct Frenchdiscord.pyjson import fix in companion PR)