Skip to content

fix(stt): better error logging and smarter DM when STT lazy-install fails - #46127

Closed
damiankluk wants to merge 4 commits into
NousResearch:mainfrom
damiankluk:main
Closed

fix(stt): better error logging and smarter DM when STT lazy-install fails#46127
damiankluk wants to merge 4 commits into
NousResearch:mainfrom
damiankluk:main

Conversation

@damiankluk

@damiankluk damiankluk commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Problem

When a user sends a voice message and faster-whisper is not installed, the gateway:

  1. Attempts a lazy-install via uv pip install
  2. If it fails (common reason: venv owned by different UID), logs the error at DEBUG level only — invisible to operators
  3. Sends a generic DM telling the user to set stt.enabled: true even when it is already true
  4. The agent gets a note saying "no STT provider is configured" with no context about why

Changes

tools/transcription_tools.py

  • _try_lazy_install_stt(): Log failures at WARNING instead of DEBUG, with the actual exception message and actionable guidance for the most common cause — the Hermes process user cannot write to the virtual environment (UID mismatch in Docker/user-namespace setups).

gateway/run.py — STT failure DM (sent to user)

  • The DM now reads the actual stt config before composing advice:
    • If stt.enabled: false → says so specifically
    • If stt.provider: local or auto-detected → shows a permission-aware install command
    • For cloud providers → suggests checking the API key
    • Falls back to the original generic message if anything fails

gateway/run.py — Agent context note

  • The agent-facing note now includes a config-aware hint about WHY transcription failed (e.g. "provider is local but faster-whisper failed to install — likely a venv write-permission issue") so the agent can give better responses.

Root cause investigated

In the reporters environment (Docker with user-namespace mapping), the gateway process runs as uid=1000 but the Hermes venv (/opt/hermes/.venv/) is owned by uid=10000. All lazy-installs fail silently because uv pip install cannot write to the venv. The .env file now includes PYTHONPATH=/home/hermes/.hermes/venv_ext pointing to a writable extension directory where faster-whisper has been pre-installed.

Testing

  • Verified that faster-whisper == 1.2.1 works when PYTHONPATH includes the extension directory
  • Transcribed a real Polish voice message successfully
  • All existing tests pass

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery tool/tts Text-to-speech and transcription labels Jun 14, 2026
tonydwb
tonydwb approved these changes Jun 15, 2026
View reviewed changes

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Improves STT lazy-install failure logging and adds smarter DM message when STT fails to install.

Looks Good

  • Better error context for debugging STT installation failures.
  • Cleaner user-facing messaging for install failures.
  • Well-scoped change.

Reviewed by Hermes Agent

@damiankluk
fix(stt): better error logging and smarter DM when STT install fails
d3e07bd 
Three improvements to the voice-transcription setup flow:

1. tools/transcription_tools.py - Log lazy-install failures at WARNING
   instead of DEBUG, and include actionable guidance about venv
   permission issues (the most common cause of silent STT failures).

2. gateway/run.py - Smart DM message: check the actual stt config
   before sending setup instructions. If stt.enabled is already true
   and provider is 'local', skip the redundant 'set stt.enabled'
   advice and show a permission-aware install hint instead.

3. gateway/run.py - Agent note now includes a config-aware hint
   about why STT is unavailable (e.g. 'provider is local but
   faster-whisper failed to install').
@damiankluk
fix(gateway): prevent zombie deadlock on /restart
9985b06 
kill -0 returns success on zombie processes (the PID still occupies
the process table), causing the restart watcher to loop forever.
Check /proc/PID/status for State:Z and bail out early. Also cap
the wait at 120 s (600 * 0.2s) as a safety net.

Fixes the /restart command hanging on Telegram when the old gateway
becomes a zombie because the dashboard (PID 1) doesn't reap children.
@damiankluk
feat(docker): include voice extra (faster-whisper) for STT support
1158430
teknium1
teknium1 reviewed Jul 14, 2026
View reviewed changes

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating the lazy-install failure path. The operator-logging concern is still present: tools/transcription_tools.py:228 logs the caught install exception at DEBUG.

Problems

  • Current main intentionally removed the hardcoded STT failure DM: gateway/run.py:10478-10486 documents that it caused duplicate, wrong-language replies. The PR restores that behavior.
  • Current main also intentionally keeps provider/setup errors out of the LLM-visible message (gateway/run.py:15258-15271), because those strings persist in conversation history. The PR's config-aware hint reverses that contract; tests/gateway/test_stt_config.py:84-104 covers the neutral-marker behavior.
  • Docker now routes lazy installs to writable durable storage (Dockerfile:300-309; tests/tools/test_dockerfile_immutable_install.py:81-110), so the /opt/hermes/.venv remediation needs redesign.

Suggested changes

  • Salvage only a tested warning-level lazy-install diagnostic, while preserving the neutral gateway message path.
  • Treat Docker packaging and the unrelated restart/workflow commits as separate current-main changes.

Automated hermes-sweeper review.

Comment thread tools/transcription_tools.py
"This is often a permission issue: the Hermes process user cannot "
"write to the virtual environment. Try running manually as the "
"venv owner: `stat -c '%%u' '$(dirname $(dirname $(which python3)))'` "
"then `su - <owner> -c 'VIRTUAL_ENV=/opt/hermes/.venv "

@teknium1 teknium1 Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This remediation hardcodes the Docker image venv, but current Docker redirects lazy installs to writable /opt/data/lazy-packages (Dockerfile:300-309). Please avoid emitting /opt/hermes/.venv commands here; keep any warning diagnostic environment-independent or derive the actual lazy-install target.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/install-update Installer, updater, packaging, wheels, doctor labels Jul 14, 2026
@teknium1 teknium1 mentioned this pull request Jul 28, 2026
Merged
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
@damiankluk @teknium1
fix(stt): better error logging when faster-whisper lazy install fails
a784e74 
Log lazy-install failures at WARNING instead of DEBUG, with actionable
guidance about venv write-permission issues (the most common cause of
silent STT failures).

Salvaged from PR #46127 (transcription_tools half only — the gateway DM
hunks are superseded by main's neutral-marker enrichment design, and the
Docker/CI files were unrelated scope).

(cherry picked from commit d3e07bd, reduced)
@teknium1

teknium1 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Merged into main via consolidated salvage PR #73510 (merge c0c5dac531). Your WARN-level logging when the faster-whisper lazy install fails was cherry-picked with your authorship (transcription_tools half; the Docker/CI hunks were superseded by main's neutral-marker design).

Your contribution is credited to you in git history. Thank you! Closing this PR as merged-via-salvage.

@teknium1 teknium1 closed this Jul 29, 2026
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@damiankluk @teknium1
fix(stt): better error logging when faster-whisper lazy install fails
5825575 
Log lazy-install failures at WARNING instead of DEBUG, with actionable
guidance about venv write-permission issues (the most common cause of
silent STT failures).

Salvaged from PR NousResearch#46127 (transcription_tools half only — the gateway DM
hunks are superseded by main's neutral-marker enrichment design, and the
Docker/CI files were unrelated scope).

(cherry picked from commit d3e07bd, reduced)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teknium1 teknium1 teknium1 left review comments
+1 more reviewer
@tonydwb tonydwb tonydwb approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-automation Sweeper risk: may affect CI, automerge, label sync, or maintainer automation sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@damiankluk @teknium1 @tonydwb @alt-glitch