Skip to content

fix(stt): preprocess .silk voice notes before transcription - #11593

Closed
dso2ng wants to merge 1 commit into
NousResearch:mainfrom
dso2ng:fix/silk-stt-compat
Closed

fix(stt): preprocess .silk voice notes before transcription#11593
dso2ng wants to merge 1 commit into
NousResearch:mainfrom
dso2ng:fix/silk-stt-compat

Conversation

@dso2ng

@dso2ng dso2ng commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR adds inbound .silk voice-note preprocessing to the STT path.

Hermes can receive voice notes from messaging platforms / bridges as Silk-encoded .silk files. Before this change, tools.transcription_tools.transcribe_audio() validated the file extension before any format-specific preprocessing, so .silk inputs failed early with Unsupported format: .silk and never reached the configured STT provider.

After this change:

  • .silk inputs are detected before normal audio-extension validation.
  • Hermes converts .silk inputs to temporary .wav files through optional pilk when available.
  • The converted .wav continues through the existing STT provider dispatch path unchanged.
  • Temporary conversion artifacts are cleaned up after transcription.
  • Already-supported audio formats keep the existing provider flow.

Related Issue

N/A — this is a focused compatibility fix for inbound Silk voice-note transcription.

Related / adjacent context:

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/transcription_tools.py
    • Adds .silk detection before normal supported-format validation.
    • Converts .silk inputs to a temporary .wav via optional pilk.
    • Sends the prepared .wav through the existing provider dispatch path.
    • Cleans up temporary conversion output after transcription.
  • tests/tools/test_transcription_tools.py
    • Adds regression coverage proving .silk inputs are converted before provider dispatch.
  • pyproject.toml / uv.lock
    • Adds the optional pilk dependency under the voice extra so Silk preprocessing is available when voice dependencies are installed.

How to Test

Focused local verification on the refreshed branch:

  1. Run the Silk preprocessing regression:

    python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o 'addopts=' -q

    Expected: 1 passed

  2. Run the affected transcription / managed-media tests:

    python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q

    Expected: 101 passed

  3. Check Python syntax for the touched transcription files:

    python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py

    Expected: no output / exit 0

  4. Verify the lockfile is current:

    uv lock --check

    Expected: exit 0

  5. Check diff whitespace:

    git diff --check

    Expected: no output / exit 0

Manual smoke test when a messaging gateway provides a Silk file:

  1. Send a Silk-encoded voice note through a messaging gateway.
  2. Confirm Hermes transcribes it instead of returning Unsupported format: .silk.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
    • Not run locally; this PR has targeted verification for the affected STT paths as listed above.
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux / local Hermes development environment

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
    • N/A; this is a narrow STT compatibility fix with regression coverage.
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
    • N/A; no config keys added or changed.
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
    • N/A; no architecture/workflow contract changed.
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
    • The implementation uses Python temporary files and the optional Python pilk dependency; no platform-specific shell commands are introduced.
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A
    • N/A; tool schema is unchanged. The accepted input path behavior now preprocesses .silk before dispatch.

For New Skills

N/A — this PR does not add a skill.

Screenshots / Logs

N/A — no UI changes.

Relevant verification output from the refreshed branch was posted in the PR discussion:

python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o 'addopts=' -q
-> 1 passed

python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q
-> 101 passed

python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
-> passed

uv lock --check
-> passed

git diff --check
-> passed

@dso2ng

dso2ng commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

Maintainer context: I did a repo search for related silk work before narrowing this PR.

Related but distinct items:

So the intended scope of this PR is specifically:

  • once a .silk file is available locally,
  • preprocess it into a form the existing STT pipeline can consume,
  • without changing the broader gateway delivery semantics.

@dso2ng

dso2ng commented Apr 21, 2026

Copy link
Copy Markdown
Contributor Author

Maintainer note: this fork-based PR already has focused local verification recorded in the PR description/comments, but GitHub Actions for the PR still appear to be stuck in action_required with no actual jobs started.

Current blocker appears to be maintainer approval for fork workflows in the GitHub UI.

If a maintainer approves / runs the PR workflows, I can help follow up on any real CI failures after the checks actually execute.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from b2fccf4 to a788f07 Compare April 21, 2026 08:36
@dso2ng

dso2ng commented Apr 21, 2026

Copy link
Copy Markdown
Contributor Author

Maintainer update: I refreshed this branch onto current main, resolved the previous merge conflict, and force-pushed a clean single-commit branch. I also switched the commit author to my GitHub noreply email, so the earlier contributor-attribution risk should be avoided once workflows actually run. The PR is now mergeable again; current blocker still appears to be fork-workflow approval (action_required / zero-job runs) in the GitHub UI.

@alt-glitch alt-glitch added type/feature New feature or request P2 Medium — degraded but workaround exists tool/tts Text-to-speech and transcription comp/tools Tool registry, model_tools, toolsets labels Apr 24, 2026
@dso2ng

dso2ng commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Update: refreshed this branch onto current main, resolved the tools/transcription_tools.py conflict while preserving the .silk preprocessing path, and pushed the updated head (fd4fe3209).

Focused verification on the refreshed branch:

  • /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o addopts= -q
  • /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m pytest tests/tools/test_transcription_tools.py -o addopts= -q

Current PR state is back to mergeable on GitHub; there are still no checks reported on the branch from the fork side.

@dso2ng
dso2ng requested a review from a team May 14, 2026 10:33
@dso2ng

dso2ng commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I merged current origin/main into this branch and resolved the pyproject.toml conflict while preserving the .silk preprocessing support. I also refreshed uv.lock so the new pilk voice-extra dependency is represented in the current lockfile.

Updated head: 440794eff

Focused local verification on the refreshed branch:

  • /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o addopts= -q -> 1 passed
  • /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o addopts= -q -> 101 passed
  • /home/dso2ng/.hermes/hermes-agent/venv/bin/python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • git diff --check

The PR is mergeable again. The current GitHub Actions runs on the refreshed head still show action_required with zero jobs, so they appear to be waiting for maintainer approval to run fork workflows.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 440794e to 4662275 Compare May 15, 2026 23:38
@dso2ng

dso2ng commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I force-pushed this branch as a clean single commit on top of current origin/main (31ba2b0cb) and resolved the previous conflicts in tools/transcription_tools.py / uv.lock.

Updated head: 466227538

The refreshed patch keeps the current main STT dispatch behavior, including local command, managed OpenAI audio gateway, and xAI paths, while adding the .silk -> .wav -> existing provider dispatch preprocessing hook. uv.lock was regenerated after adding the optional pilk voice-extra dependency.

Focused local verification on the refreshed branch:

  • python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o 'addopts=' -q -> 1 passed
  • python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q -> 101 passed
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • git diff --check

GitHub metadata now reports the PR as mergeable again. There are still no checks reported on this fork branch, so CI appears not to have run yet.

@austinpickett
austinpickett requested a review from Copilot May 19, 2026 12:38

@austinpickett austinpickett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use .github/PULL_REQUEST_TEMPLATE.md

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@dso2ng

dso2ng commented May 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review note. I updated the PR description to follow .github/PULL_REQUEST_TEMPLATE.md while preserving the focused scope, affected paths, and targeted verification details. No code changes were made in this update.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 4662275 to e272166 Compare May 25, 2026 03:38
@dso2ng

dso2ng commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I force-pushed this branch as a clean single commit on top of current origin/main.

Updated head: e27216608

This also keeps the PR description in the current .github/PULL_REQUEST_TEMPLATE.md structure to address the prior review note. No inline review comments are currently outstanding.

Focused local verification on the refreshed branch:

  • python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o 'addopts=' -q -> 1 passed
  • python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q -> 101 passed
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py -> passed
  • local equivalent of Contributor Attribution Check -> passed with author 275702+dso2ng@users.noreply.github.com
  • uv lock --check -> passed
  • git diff --check -> passed

GitHub Actions on the new head are still action_required with zero check runs, so they appear to be waiting for maintainer approval to run fork workflows. Once workflows are approved, I can follow up on any real CI failures.

@ryan-flow

Copy link
Copy Markdown

test

@ryan-flow

Copy link
Copy Markdown

Hi @dso2ng, I'd like to help get this PR across the finish line. The collaborator requested using .github/PULL_REQUEST_TEMPLATE.md — would you be open to me forking your branch and opening a fresh PR with the updated template format? Or if you'd prefer, I can help address the review feedback on this PR directly.

I'm also interested in extending this to cover the outbound direction (OGG → SILK for voice bubbles, #9971) once the inbound preprocessing lands.

@dso2ng

dso2ng commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

cc @austinpickett — the template feedback has been addressed and the branch is mergeable again.

Thanks @ryan-flow — I appreciate the offer to help.

I refreshed this PR directly rather than opening a duplicate fresh PR, since the description is already in the .github/PULL_REQUEST_TEMPLATE.md structure now.

Updated head: 416b0e9e9

What changed in this refresh:

  • Rebasing/cherry-picking onto current origin/main made the branch mergeable again.
  • Resolved the tools/transcription_tools.py conflict by keeping the newer command/plugin STT provider dispatch from main while preserving this PR's .silk -> .wav -> existing provider dispatch preprocessing path.
  • All STT dispatch paths now use the prepared audio path after preprocessing, and the temporary conversion directory is still cleaned up in finally.

Focused verification on the refreshed branch:

  • python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch -o 'addopts=' -q -> 1 passed
  • python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q -> 102 passed
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py -> passed
  • uv lock --check -> passed
  • git diff --check -> passed
  • public diff non-ASCII guard -> non_ascii_added_lines=0

The PR is mergeable again. GitHub still reports no checks on this fork branch, so the next blocker appears to be the usual fork-workflow approval / re-review rather than a local test failure.

For #9971 / outbound OGG->SILK voice bubbles: agreed that it should stay as a separate follow-up from this inbound preprocessing PR, so #32212 looks like the right place for that direction.

@dso2ng

dso2ng commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

Following up on the earlier template review request:

  • The PR description has been updated to use .github/PULL_REQUEST_TEMPLATE.md.
  • There are no outstanding inline review comments on the PR at the moment.
  • The current head is still mergeable against main (416b0e9e).
  • The latest workflow runs are still action_required with zero jobs, so CI appears to be waiting for maintainer approval to run fork workflows rather than failing on the branch.

Could a maintainer please re-review / clear the stale template-related “changes requested” state and approve the fork workflow runs when you have a chance? I’m happy to follow up on any real CI failures once the checks are allowed to run.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 416b0e9 to 9b7a2d2 Compare May 31, 2026 23:38
@dso2ng

dso2ng commented May 31, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I rebased this branch onto current origin/main and resolved the tools/transcription_tools.py conflict. The refreshed version keeps the current upstream ElevenLabs STT path while preserving this PR's .silk preprocessing and temporary-file cleanup behavior.

Updated head: 9b7a2d2fd

Focused local verification on the refreshed head:

  • git diff --check origin/main...HEAD
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • python -m pytest tests/tools/test_transcription_tools.py -q -> 108 passed
  • scripts/run_tests.sh tests/tools/test_transcription_tools.py -> 108 tests passed
  • non-ASCII diff guard -> non_ascii_added_lines=0

GitHub Actions is still blocked at the fork workflow approval gate (action_required, zero jobs on the new head), so there are no CI logs to debug yet.

cc @austinpickett when you have a chance.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 9b7a2d2 to 53234c5 Compare June 2, 2026 23:53
@dso2ng

dso2ng commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I rebased this branch onto current origin/main and resolved the uv.lock conflict while preserving the current upstream lock additions and this PR's .silk preprocessing dependency.

Updated head: 53234c5ae

Targeted local verification on the refreshed head:

  • git diff --check origin/main...HEAD
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • python -m pytest tests/tools/test_transcription_tools.py -o 'addopts=' -q108 passed
  • non-ASCII diff guard → non_ascii_added_lines=0

GitHub Actions is still at the fork workflow approval gate on the new head (action_required with zero jobs), so no real CI logs are available yet.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 53234c5 to 4d4f86a Compare June 5, 2026 00:12
@dso2ng

dso2ng commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I rebased this branch onto current origin/main and resolved the uv.lock conflict while preserving current upstream lock entries plus this PR's .silk preprocessing dependency.

Updated head: 4d4f86a68

Targeted local verification on the refreshed head:

  • git diff --check origin/main...HEAD
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • scripts/run_tests.sh tests/tools/test_transcription_tools.py108 tests passed
  • non-ASCII diff guard → non_ascii_added_lines=0

GitHub metadata now reports the PR as mergeable again. The new workflow runs are still blocked at the fork workflow approval gate (action_required with zero jobs), so there are no real CI logs to debug yet.

cc @austinpickett when you have a chance to re-review / clear the stale template-related changes-requested state.

@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 4d4f86a to 0be5d78 Compare June 7, 2026 12:41
@dso2ng

dso2ng commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Refresh update: I rebased this branch onto current origin/main and resolved the uv.lock conflict by keeping the current upstream core pillow lock entry while preserving this PR's pilk voice extra.

Updated head: 0be5d78ba

Targeted local verification on the refreshed head:

  • git diff --check origin/main...HEAD
  • python -m py_compile tools/transcription_tools.py tests/tools/test_transcription_tools.py
  • uv lock --check
  • uv run --frozen --extra dev python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q120 passed
  • non-ASCII diff guard → non_ascii_added_lines=0

GitHub metadata now reports the PR as mergeable again. The new workflow runs are still at the fork workflow approval gate (action_required with zero jobs), so there are no real CI logs to debug yet.

cc @austinpickett when you have a chance to re-review / clear the stale template-related changes-requested state.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused inbound-STT compatibility work. The premise is valid: current main rejects .silk before provider dispatch (tools/transcription_tools.py:1032-1037, 1643-1646).

Problems

  • tools/transcription_tools.py:1679 preprocesses before validation, and the helper invokes pilk.silk_to_wav() at line 1070. This permits .silk symlinks to reach the decoder before current main's symlink rejection at line 1026. Commit 95848b1cbcf3549490cc413b701080d26d33e0f5 explicitly established that validation must precede dispatch.
  • The same order means the original .silk bypasses the current 25 MiB limit (tools/transcription_tools.py:1038-1045) until after conversion; the PR validates only prepared_path at line 1691.
  • The new test at tests/tools/test_transcription_tools.py:921 mocks both preprocessing and validation, so it cannot cover either regression.

Suggested changes

  • Validate source path safety and source size before decoding, while deferring only extension validation until the converted WAV exists.
  • Add .silk symlink and oversized-source tests that assert the converter is not called.

Automated hermes-sweeper review.

Comment thread tools/transcription_tools.py Outdated
"error": "STT is disabled in config.yaml (stt.enabled: false).",
}
try:
prepared_path, cleanup_dir, prep_error = _prepare_audio_for_transcription(file_path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This invokes the decoder before the existing validation. A .silk symlink reaches pilk.silk_to_wav() before _validate_audio_file() can reject it, regressing the explicit symlink hardening from 95848b1. Split source safety/size checks from extension validation and run the former before this call.

@@ -909,6 +917,24 @@ def test_model_override_passed_to_local(self, sample_ogg):

assert mock_local.call_args[0][1] == "large-v3"

def test_converts_silk_before_dispatch(self, sample_silk):
with patch("tools.transcription_tools._prepare_audio_for_transcription",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this test mocks both preprocessing and _validate_audio_file, it cannot detect the new decoder-before-validation ordering. Add regression coverage for a .silk symlink and an oversized .silk source that asserts preprocessing is not invoked.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
@dso2ng
dso2ng force-pushed the fix/silk-stt-compat branch from 0be5d78 to e5db793 Compare July 13, 2026 02:29
@dso2ng

dso2ng commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review — addressed on the current head e5db79369.

  • Source safety and source-size validation now run before any .silk preprocessing/decoder invocation; extension validation is deferred until the converted WAV exists.
  • Added regression coverage for both a .silk symlink and an oversized .silk source. Each asserts that preprocessing is not called.

Focused verification on the updated head:

python -m pytest tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_converts_silk_before_dispatch tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_silk_symlink_is_rejected_before_preprocessing tests/tools/test_transcription_tools.py::TestTranscribeAudioDispatch::test_oversized_silk_is_rejected_before_preprocessing -o 'addopts=' -q
3 passed

python -m pytest tests/tools/test_transcription_tools.py tests/tools/test_managed_media_gateways.py -o 'addopts=' -q
124 passed

Could you please re-review the updated head?

@teknium1

Copy link
Copy Markdown
Contributor

Merged into main via consolidated salvage PR #73510 (merge c0c5dac531). Your .silk (WeChat/QQ) voice-note preprocessing was cherry-picked with your authorship, reworked so pilk is a lazy dep (stt.silk) and the decode lives centrally in transcribe_audio.

Your contribution is credited to you in git history. Thank you! Closing this PR as merged-via-salvage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants