feat: add API server audio endpoints - #29364
Codename-11 wants to merge 1 commit into
Conversation
alt-glitch
added
type/feature
|
Duplicate of #8199 which adds the same |
|
Good catch, I missed #8199, apologies! Looks like that already covers the If #8199 is the preferred base, I can close this, or rework it into a smaller discovery/compat follow-up. 😁 |
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks for building this on the existing STT/TTS primitives. The current-main premise remains valid: gateway/platforms/api_server.py:1546 advertises audio_api: false, and its route registration at gateway/platforms/api_server.py:4843-4883 has no audio route.
Problems
gateway/platforms/api_server.py:1121breaks out after findingfile, so a multipartmodelfield sent afterfileis ignored.gateway/platforms/api_server.py:1195invokes TTS withoutoutput_path; the shared tool creates an output artifact when none is provided (tools/tts_tool.py:2204-2247), but the handler only reads it at:1210and never deletes it.gateway/platforms/api_server.py:1216always returnsaudio/mpeg, although the TTS primitive returns an artifact path and supports non-MP3 output (tools/tts_tool.py:2440-2446).- The API surface lacks corresponding documentation in
website/docs/user-guide/features/api-server.md.
Suggested changes
- Coordinate/consolidate with the overlapping open #8199 before selecting a canonical public contract.
- Parse all multipart fields, use a temporary TTS output with guaranteed cleanup, detect the artifact MIME type, and add regression coverage for each case.
Automated hermes-sweeper review.
| continue | ||
| if part.name in {"file", "audio"}: | ||
| file_field = part | ||
| break |
There was a problem hiding this comment.
Breaking here makes multipart field order observable: a normal request with file before model never forwards the requested model. Stream the file while continuing through remaining parts, and add a file-before-model regression test.
|
|
||
| from tools.tts_tool import text_to_speech_tool | ||
|
|
||
| result_raw = await asyncio.to_thread(text_to_speech_tool, text=text.strip()) |
There was a problem hiding this comment.
Without output_path, text_to_speech_tool creates its normal persistent output artifact. This handler reads it but never deletes it, so each successful API request leaks a generated audio file. Use a dedicated temporary output path and clean it up after responding.
| return web.json_response(_openai_error("Failed to read generated speech audio", code="tts_read_failed"), status=502) | ||
|
|
||
| headers = {"X-Hermes-TTS-Provider": str(result.get("provider") or "")} | ||
| return web.Response(body=audio_bytes, content_type="audio/mpeg", headers=headers) |
There was a problem hiding this comment.
The shared TTS layer returns the actual artifact path and providers may produce WAV, OGG, or other output. Always declaring audio/mpeg can mislabel the response; detect the generated artifact MIME type or constrain and verify the requested output format.
teknium1
added
sweeper:risk-security-boundary
3 participants
Summary
GET /api/audio/capabilities,POST /api/audio/transcriptions, andPOST /api/audio/speech./v1/capabilitieswith fallback-safe endpoint metadata.tools.transcription_tools.transcribe_audioandtools.tts_tool.text_to_speech_tool) instead of introducing a parallel provider layer./voice/*compatibility aliases for existing Relay-style clients.Deferred Scope
Test Plan
python -m py_compile gateway/platforms/api_server.py tests/gateway/test_api_server.pypython -m pytest tests/gateway/test_api_server.py::TestAudioEndpoints -q -o 'addopts='python -m pytest tests/gateway/test_api_server.py -q -o 'addopts='python -m pytest tests/gateway/test_api_server_runs.py tests/gateway/test_api_server_jobs.py -q -o 'addopts='git diff --check