Skip to content

STT v3 — route hermes-agent to whisper.service HTTP + cloud OpenAI primary - #3

Merged
MKheru merged 1 commit into
acos-mainfrom
ah/stt-v3-migration-2026-05-03
May 3, 2026
Merged

STT v3 — route hermes-agent to whisper.service HTTP + cloud OpenAI primary#3
MKheru merged 1 commit into
acos-mainfrom
ah/stt-v3-migration-2026-05-03

Conversation

@MKheru

@MKheru MKheru commented May 3, 2026

Copy link
Copy Markdown
Owner

Summary

Migrates the STT path used by hermes-agent for inbound voice messages from faster-whisper base (in-process, ~39M params) to:

  1. Cloud primary: OpenAI gpt-4o-mini-transcribe (sub-second, $0.003/min, no hallucinations on 4-audio bench)
  2. Local fallback: whisper.service HTTP at http://127.0.0.1:8765/inference (whisper.cpp server already running with ggml-large-v3-turbo)

The local fallback closes a longstanding mismatch: whisper.service was already deployed with the large-v3-turbo model but hermes-agent silently loaded the much smaller base model in-process via faster-whisper.

What's in the diff

tools/transcription_tools.py

  • New _transcribe_via_whisper_service(file_path, endpoint, language) function. Pure stdlib urllib (no requests dep), manually-constructed multipart form-data, OpenAI-compatible response shape. Returns same dict as legacy _transcribe_local().
  • _transcribe_local() now tries the HTTP endpoint first (when stt.local.endpoint is configured), falls back to in-process faster-whisper on any HTTP failure.

tools/voice_io.py (newly tracked)

  • WS16 voice loop client. Talks to whisper.service (STT) and tts.service (TTS) via HTTP. No subprocess (SECCOMP-compatible).

uv.lock

  • Auto-updated by uv during the session.

Validation

Bench performed by Claude on 4 real audios (Khéri's Discord vocals, archived at ~/.hermes/cache/audio/):

Backend Latency Hallucinations Cost
whisper-1 (cloud) 1.2–4.6s 1/4 totalement halluciné ("Merci Philip..." + leak chinois) $0.006/min
gpt-4o-mini-transcribe (cloud) 0.7–2.3s 0/4 $0.003/min
whisper.service large-v3-turbo (local) 23–47s (CPU only) 0/4 $0

gpt-4o-mini-transcribe retained as primary. whisper-1 disqualified on hallucination rate. whisper.service retained as fallback (slow but free + already deployed).

User-facing config

Requires ~/.hermes/config.yaml to be updated to set:

stt:
  provider: openai
  local:
    endpoint: http://127.0.0.1:8765/inference
  openai:
    model: gpt-4o-mini-transcribe

Config not in this PR (it's user runtime, not in repo). Applied to prod separately.

Test plan

  • Smoke test: _transcribe_via_whisper_service against archived OGG → success + correct French
  • Live test: Discord voice message transcribed end-to-end via new route
  • Native voice reply path tested (depends on discord.py json import fix in companion PR)

…I primary

Fixes the local STT mismatch where hermes-agent loaded faster-whisper 'base'
in-process while whisper.service was already running with ggml-large-v3-turbo
on port 8765 — the better model was deployed but never used.

tools/transcription_tools.py:
  - New _transcribe_via_whisper_service() that POSTs OGG audio to a
    configurable HTTP endpoint (whisper.cpp server). Multipart manually
    constructed via stdlib urllib so no new pip dep, and works under
    SECCOMP without subprocess.
  - _transcribe_local() now tries the HTTP endpoint first (when configured
    via stt.local.endpoint), falls back to faster-whisper in-process.

tools/voice_io.py:
  - Tracked for the first time. Implements the WS16 voice loop client
    talking to whisper.service (STT 8765) and tts.service (TTS 8766) via
    HTTP. Used by the Discord adapter for STT on inbound vocals and TTS
    on outbound native voice messages. Pure stdlib + aiohttp; no
    subprocess (compatible with hermes-agent.service SECCOMP).

uv.lock:
  - Auto-updated by uv during the session; included for reproducibility.

User-facing config note: ~/.hermes/config.yaml stt.local.endpoint is a new
optional key. When set (e.g. http://127.0.0.1:8765/inference) the
transcription router routes local calls to whisper.service first. When
unset, the legacy faster-whisper in-process path is preserved.

Bench validation (Claude, 2026-05-03 on 4 archived audios):
  whisper-1 cloud:           1.2-4.6s, 1/4 hallucinations (disqualified)
  gpt-4o-mini-transcribe:    0.7-2.3s, 0/4 hallucinations (recommended)
  whisper.service L3T local: 23-47s,    0/4 hallucinations (good fallback)

Smoke test post-deploy: _transcribe_via_whisper_service returns
success=True with correct French transcript via large-v3-turbo. Live
vocal at 18:35 UTC transcribed end-to-end through the new route.
@MKheru
MKheru merged commit 44fa7cb into acos-main May 3, 2026
@MKheru
MKheru deleted the ah/stt-v3-migration-2026-05-03 branch May 3, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant