feat(tools): forward OpenAI TTS instructions field through text_to_speech - #14205
Closed
0xAlcibiades wants to merge 3 commits into
Closed
0xAlcibiades wants to merge 3 commits into
0xAlcibiades wants to merge 3 commits into
Conversation
…eech The `text_to_speech` tool schema accepted only `text` and `output_path`, so style direction (tone, emotion, pacing, whispering) could never reach the OpenAI backend — even though `gpt-4o-mini-tts` (Hermes's OpenAI provider default) treats `instructions` as its primary voice-design control. This plumbs an optional `instructions` argument through the tool schema, the handler lambda, and `text_to_speech_tool()` into `_generate_openai_tts`, where it is forwarded to `client.audio.speech.create()` only when truthy. Empty/None values still omit the key entirely, preserving behavior on `tts-1`/`tts-1-hd` and strict OpenAI-compatible servers. The same passthrough unblocks self-hosted OpenAI-compatible voice-design servers (Qwen3-TTS-VoiceDesign on oMLX, etc.) that are already wired in via `tts.openai.base_url` — the established convention per NousResearch#9004 and the TTS config docs — without inventing a new provider backend. Tests: `tests/tools/test_tts_instructions.py` covers backend passthrough, tool-level threading, schema declaration, and the empty-string/absent omission cases. `tests/tools/test_tts_max_text_length.py` fake_openai signature widened to accept the new kwarg. Refs NousResearch#14196
0xAlcibiades
force-pushed
the
feat/tts-openai-instructions
branch
from
April 23, 2026 00:04
e080ced to
f109c3f
Compare
alt-glitch
added
type/feature
Closed
Contributor
Author
|
Any interest in getting this merged? |
teknium1
reviewed
Jul 12, 2026
teknium1
left a comment
teknium1
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the focused TTS extension. The premise remains valid on current main: tools/tts_tool.py:1059-1068 constructs the OpenAI request without instructions, while tools/tts_tool.py:2153-2156 and :2847-2867 expose and dispatch only text and output_path.
Problems
- The two new mocks in
tests/tools/test_tts_instructions.py:33and:86return two values from_resolve_openai_audio_client_config(). Current main now unpacks three values attools/tts_tool.py:1027, introduced byb53ba0e188363800447b7aca4c0a85420c39d1a8; these tests will fail before reaching their assertions.
Suggested changes
- Return
("test-key", None, False)from both mocks. The production change otherwise follows the existingspeedconditional-kwarg pattern attools/tts_tool.py:1059-1068, and the schema tests correctly keep the new field optional.
Automated hermes-sweeper review.
|
|
||
| with patch("tools.tts_tool._import_openai_client", return_value=mock_cls), \ | ||
| patch("tools.tts_tool._resolve_openai_audio_client_config", | ||
| return_value=("test-key", None)): |
Collaborator
There was a problem hiding this comment.
Current main now unpacks three values from this helper (tools/tts_tool.py:1027, commit b53ba0e188363800447b7aca4c0a85420c39d1a8). Return ("test-key", None, False) here and in the second helper so these tests do not fail during setup.
teknium1
added
sweeper:risk-compatibility
Collaborator
|
Merged into main via consolidated salvage PR #73513 (merge Your contribution is credited to you in git history. Thank you! Closing this PR as merged-via-salvage. |
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14196.
Summary
instructionsparameter to thetext_to_speechtool schema and threads it throughtext_to_speech_tool()→_generate_openai_tts()→client.audio.speech.create().tts-1/tts-1-hdand strict OpenAI-compatible servers that reject unknown kwargs are unaffected.gpt-4o-mini-tts's voice-design capability (tone, emotion, pacing, accent, whispering) — already supported by the model but previously unreachable through the tool.tts.openai.base_url— the established convention per [Bug]: TTS truncates at first newline — Kokoro and potentially other backends #9004.Docs:
voice-mode.mdupdated with a short note under thetts.openaiblock and a line in the TTS Provider Comparison.Why
gpt-4o-mini-ttsis Hermes's default model on theopenaiTTS provider, and itsinstructionsfield (OpenAI docs) is the headline quality lever on that model. The hard-coded kwarg list in_generate_openai_ttssilently dropped any style direction, so every reply in voice mode got the same flat default regardless of what was being said. This PR is the minimum change to expose that capability.Test plan
New file
tests/tools/test_tts_instructions.py(6 tests) covers:_generate_openai_ttsforwardsinstructionstoaudio.speech.createwhen providedinstructionskey absent from create kwargs when not provided (regression guard fortts-1/ strict servers)instructionsomitted (treated as absent)text_to_speech_tool(instructions=...)→ backend sees itinstructionsas optional string, not inrequiredExisting
tests/tools/test_tts_max_text_length.pyfake helper widened to accept the new kwarg.Run locally:
uv run scripts/run_tests.sh tests/tools/test_tts_instructions.py tests/tools/test_tts_speed.py tests/tools/test_tts_max_text_length.py # 50 passedManual verification against
gpt-4o-mini-tts: same text with vs. withoutinstructions="Whisper conspiratorially"produces audibly different output. Also verified against a self-hosted oMLX server servingQwen3-TTS-12Hz-1.7B-VoiceDesign-bf16viatts.openai.base_urloverride.Platforms tested
No platform-specific code touched; change is pure kwarg passthrough in the existing OpenAI client call.