feat(tts): add voice upload API for Qwen3-TTS#1201
Conversation
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e405d2eff
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # Validate Base task requirements | ||
| if task_type == "Base": | ||
| if task_type == "Base" and request.voice is None: | ||
| if request.ref_audio is None: | ||
| return "Base task requires 'ref_audio' for voice cloning" |
There was a problem hiding this comment.
The new Base-task validation only enforces ref_audio when voice is missing, so a request like task_type=Base with a built-in speaker name but no ref_audio now passes validation. In that case _build_tts_params will send no ref_audio to the model (because the auto-fill only happens for uploaded voices), which breaks the Base task’s voice-cloning requirement and likely yields a model error or incorrect output. Consider requiring ref_audio unless voice refers to an uploaded speaker that will be auto-populated.
Useful? React with 👍 / 👎.
| # Generate filename | ||
| timestamp = int(time.time()) | ||
| file_ext = audio_file.filename.split('.')[-1] if '.' in audio_file.filename else "wav" | ||
| filename = f"{name}_{consent}_{timestamp}.{file_ext}" | ||
| file_path = self.uploaded_speakers_dir / filename | ||
|
|
There was a problem hiding this comment.
The upload endpoint builds filename directly from untrusted name and consent and then writes file_path = self.uploaded_speakers_dir / filename. If either field contains path separators or .., the resulting path can escape /tmp/voice_samples and overwrite arbitrary files on the host. This is a security issue that can be triggered by a client POSTing a crafted name/consent. Sanitize these inputs (e.g., allowlist safe characters) or normalize and validate that the resolved path stays within the upload directory.
Useful? React with 👍 / 👎.
|
A few thoughts: (1) Consider supporting pre-extracted embedding uploads (.pt/.npy) in addition to audio files to skip extraction overhead at inference time. (2) The /tmp/voice_samples storage is volatile. Maybe make this path configurable or document this limitation. (3) Missing a DELETE endpoint to remove uploaded voices. |
|
please also update the docs as well in apiserver |
we probably need to merge #1206 first |
There was a problem hiding this comment.
Pull request overview
This pull request adds voice upload functionality for Qwen3-TTS, allowing users to upload custom voice samples for voice cloning. The implementation adds new API endpoints for uploading and listing voice samples, along with automatic integration into the TTS workflow.
Changes:
- Added POST /v1/audio/voices endpoint for uploading custom voice samples (max 10MB)
- Modified GET /v1/audio/voices endpoint to return both built-in and uploaded voices
- Implemented auto-set behavior that automatically uses uploaded voice audio for Base task TTS requests
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 20 comments.
| File | Description |
|---|---|
| vllm_omni/entrypoints/openai/serving_speech.py | Core voice upload logic including file storage, metadata management, and auto-set ref_audio behavior for uploaded voices |
| vllm_omni/entrypoints/openai/api_server.py | API endpoint definitions for voice upload and enhanced voice listing with uploaded voice details |
| examples/online_serving/qwen3_tts/README.md | Documentation for new voice management endpoints with usage examples |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| # Validate Base task requirements | ||
| if task_type == "Base": | ||
| if task_type == "Base" and request.voice is None: |
There was a problem hiding this comment.
The validation doesn't check if an uploaded voice file actually exists when using Base task with an uploaded voice. If task_type is "Base" and voice is an uploaded voice name, but the audio file is missing or unreadable, the auto-set logic at lines 320-325 will silently fail (returning None from _get_uploaded_audio_data), and the Base task will proceed without ref_audio, potentially causing downstream errors. Consider adding validation to ensure uploaded voices have accessible audio files, especially for Base task.
| if task_type == "Base" and request.voice is None: | |
| if task_type == "Base": | |
| # Base task always requires explicit ref_audio to avoid relying on | |
| # potentially failing auto-set logic from uploaded voices. |
| def __init__(self, *args, **kwargs): | ||
| super().__init__(*args, **kwargs) | ||
| # Initialize uploaded speakers storage | ||
| self.uploaded_speakers_dir = Path("/tmp/voice_samples") |
There was a problem hiding this comment.
Using a hardcoded path '/tmp/voice_samples' poses several issues:
- Security: Multiple users/deployments on the same system will share this directory
- Persistence: Files in /tmp may be deleted by system cleanup processes
- Portability: This path may not work on all operating systems (e.g., Windows)
Consider using a configurable directory path that can be set via environment variable or configuration parameter, and ensure proper isolation for multi-tenant scenarios.
| self.uploaded_speakers_dir = Path("/tmp/voice_samples") | |
| base_dir_env = os.getenv("VLLM_OMNI_VOICE_SAMPLES_DIR") | |
| if base_dir_env: | |
| self.uploaded_speakers_dir = Path(base_dir_env) | |
| else: | |
| # Use a portable, user-specific cache directory by default | |
| xdg_cache_home = os.getenv("XDG_CACHE_HOME") | |
| if xdg_cache_home: | |
| cache_base = Path(xdg_cache_home) | |
| else: | |
| cache_base = Path.home() / ".cache" | |
| self.uploaded_speakers_dir = cache_base / "vllm_omni" / "voice_samples" |
|
|
||
| # Save audio file | ||
| try: | ||
| with open(file_path, 'wb') as f: | ||
| content = await audio_file.read() | ||
| f.write(content) | ||
| except Exception as e: | ||
| raise ValueError(f"Failed to save audio file: {e}") | ||
|
|
||
| # Update metadata | ||
| self.uploaded_speakers[voice_name_lower] = { | ||
| "name": name, | ||
| "consent": consent, | ||
| "file_path": str(file_path), | ||
| "created_at": timestamp, | ||
| "mime_type": mime_type, | ||
| "original_filename": audio_file.filename, | ||
| "file_size": file_size | ||
| } | ||
|
|
||
| # Update supported speakers | ||
| self.supported_speakers.add(voice_name_lower) | ||
|
|
||
| # Save metadata | ||
| self._save_uploaded_speakers() | ||
|
|
||
| logger.info(f"Uploaded new voice '{name}' with consent ID '{consent}'") | ||
|
|
||
| return { | ||
| "name": name, | ||
| "consent": consent, | ||
| "file_path": str(file_path), | ||
| "created_at": timestamp, | ||
| "mime_type": mime_type, | ||
| "file_size": file_size |
There was a problem hiding this comment.
There's a potential race condition: if the file is successfully written but saving metadata fails, the uploaded file becomes orphaned. Consider using a transaction-like pattern where you first save the file with a temporary name, then update metadata, and only rename to final name if both succeed. Also consider cleanup of orphaned files on initialization.
| # Save audio file | |
| try: | |
| with open(file_path, 'wb') as f: | |
| content = await audio_file.read() | |
| f.write(content) | |
| except Exception as e: | |
| raise ValueError(f"Failed to save audio file: {e}") | |
| # Update metadata | |
| self.uploaded_speakers[voice_name_lower] = { | |
| "name": name, | |
| "consent": consent, | |
| "file_path": str(file_path), | |
| "created_at": timestamp, | |
| "mime_type": mime_type, | |
| "original_filename": audio_file.filename, | |
| "file_size": file_size | |
| } | |
| # Update supported speakers | |
| self.supported_speakers.add(voice_name_lower) | |
| # Save metadata | |
| self._save_uploaded_speakers() | |
| logger.info(f"Uploaded new voice '{name}' with consent ID '{consent}'") | |
| return { | |
| "name": name, | |
| "consent": consent, | |
| "file_path": str(file_path), | |
| "created_at": timestamp, | |
| "mime_type": mime_type, | |
| "file_size": file_size | |
| temp_file_path = self.uploaded_speakers_dir / f"{filename}.tmp" | |
| # Save audio file to a temporary path first to avoid orphaned files | |
| try: | |
| content = await audio_file.read() | |
| with open(temp_file_path, "wb") as f: | |
| f.write(content) | |
| # Update metadata in memory | |
| self.uploaded_speakers[voice_name_lower] = { | |
| "name": name, | |
| "consent": consent, | |
| "file_path": str(file_path), | |
| "created_at": timestamp, | |
| "mime_type": mime_type, | |
| "original_filename": audio_file.filename, | |
| "file_size": file_size, | |
| } | |
| # Update supported speakers | |
| self.supported_speakers.add(voice_name_lower) | |
| # Persist metadata | |
| self._save_uploaded_speakers() | |
| # Atomically move the temp file to its final location | |
| os.replace(temp_file_path, file_path) | |
| except Exception as e: | |
| # Clean up temp file and roll back in-memory state on failure | |
| try: | |
| if isinstance(temp_file_path, Path): | |
| if temp_file_path.exists(): | |
| temp_file_path.unlink() | |
| else: | |
| if os.path.exists(temp_file_path): | |
| os.remove(temp_file_path) | |
| except Exception: | |
| # Best-effort cleanup; ignore secondary errors | |
| pass | |
| # Roll back any partially updated metadata | |
| if hasattr(self, "uploaded_speakers"): | |
| self.uploaded_speakers.pop(voice_name_lower, None) | |
| if hasattr(self, "supported_speakers"): | |
| try: | |
| self.supported_speakers.discard(voice_name_lower) | |
| except AttributeError: | |
| # In case supported_speakers is not a set-like object | |
| try: | |
| self.supported_speakers.remove(voice_name_lower) | |
| except Exception: | |
| pass | |
| raise ValueError(f"Failed to upload voice: {e}") | |
| logger.info(f"Uploaded new voice '{name}' with consent ID '{consent}'") | |
| return { | |
| "name": name, | |
| "consent": consent, | |
| "file_path": str(file_path), | |
| "created_at": timestamp, | |
| "mime_type": mime_type, | |
| "file_size": file_size, |
| def _save_uploaded_speakers(self) -> None: | ||
| """Save uploaded speakers to metadata file.""" | ||
| try: | ||
| metadata = {"uploaded_speakers": self.uploaded_speakers} | ||
| with open(self.metadata_file, 'w') as f: | ||
| json.dump(metadata, f, indent=2) | ||
| except Exception as e: | ||
| logger.error(f"Could not save uploaded speakers metadata: {e}") |
There was a problem hiding this comment.
The metadata.json file could grow unbounded as users upload more voices. There's no mechanism to limit the number of uploaded voices or to delete old voices. Consider implementing:
- A maximum number of uploaded voices per instance
- An API endpoint to delete uploaded voices
- A cleanup mechanism for old/unused voices
| def _save_uploaded_speakers(self) -> None: | ||
| """Save uploaded speakers to metadata file.""" | ||
| try: | ||
| metadata = {"uploaded_speakers": self.uploaded_speakers} | ||
| with open(self.metadata_file, 'w') as f: | ||
| json.dump(metadata, f, indent=2) | ||
| except Exception as e: | ||
| logger.error(f"Could not save uploaded speakers metadata: {e}") |
There was a problem hiding this comment.
The metadata file is not protected by any locking mechanism. In a multi-process or multi-threaded environment, concurrent uploads could lead to race conditions where:
- Two processes read the same metadata
- Both add their voice
- One overwrites the other's changes when saving
Consider using file locking (e.g., fcntl on Unix, msvcrt on Windows) or a database for thread-safe metadata storage.
| @@ -1,7 +1,11 @@ | |||
| import asyncio | |||
| import json | |||
| import os | |||
There was a problem hiding this comment.
The 'os' module is imported but never used in the code. This import should be removed to keep the codebase clean.
| import os |
| "voice": { | ||
| "name": "custom_voice_1", | ||
| "consent": "user_consent_id", | ||
| "file_path": "/tmp/voice_samples/custom_voice_1_user_consent_id_1738660000.wav", |
There was a problem hiding this comment.
The documentation exposes the internal file path '/tmp/voice_samples/' in the response example. This is a potential information disclosure issue as it reveals the server's internal directory structure. Consider either:
- Not returning the file_path in the API response
- Sanitizing the path to not reveal absolute server paths
- Returning a relative or opaque identifier instead
| "file_path": "/tmp/voice_samples/custom_voice_1_user_consent_id_1738660000.wav", | |
| "file_path": "custom_voice_1_user_consent_id_1738660000.wav", |
|
|
||
| # Generate filename | ||
| timestamp = int(time.time()) | ||
| file_ext = audio_file.filename.split('.')[-1] if '.' in audio_file.filename else "wav" |
There was a problem hiding this comment.
The file extension extraction logic is fragile. If the filename has no extension or multiple dots (e.g., 'my.voice.sample.wav'), splitting by '.' and taking the last element works, but if there's no dot in the filename, the entire filename becomes the extension. This should be handled more robustly, perhaps by using Path(audio_file.filename).suffix or providing a default extension if none is found.
| file_ext = audio_file.filename.split('.')[-1] if '.' in audio_file.filename else "wav" | |
| raw_filename = audio_file.filename or "" | |
| suffix = Path(raw_filename).suffix.lstrip(".") | |
| file_ext = suffix if suffix else "wav" |
| consent: str = Form(...), | ||
| name: str = Form(...), |
There was a problem hiding this comment.
The consent parameter is stored but never validated or used for any authorization checks. If consent is meant to represent user consent for voice cloning, there should be validation logic to verify:
- The consent ID format/validity
- Whether the consent is still active
- Logging/audit trail for consent usage
Without proper consent validation, this could lead to compliance issues with privacy regulations.
|
|
||
| #### POST /v1/audio/voices | ||
|
|
||
| Upload a new voice sample for voice cloning in Base task TTS requests. |
There was a problem hiding this comment.
The documentation states that uploaded voices can be used "for voice cloning in Base task TTS requests", but the implementation doesn't enforce that uploaded voices are only used with Base task. An uploaded voice can be used with any task type due to the auto-set logic at lines 320-325, which could lead to unexpected behavior. Consider either:
- Clarifying in the documentation that uploaded voices work with any task type
- Restricting uploaded voices to Base task only in the code
- Making the auto-set behavior conditional on task_type being "Base"
| Upload a new voice sample for voice cloning in Base task TTS requests. | |
| Upload a new voice sample that can be used for voice cloning in subsequent TTS requests with any supported task type. |
already add docs in apiserver, copy from #1206 |
|
A few issues from the Copilot review still look unaddressed after the latest commit: Security (must fix before merge):
Logic bugs (must fix):
Minor (nice to have):
Also heads up: PR #1227 adds |
you are right,i will fix it |
Port the voice upload API (POST /v1/audio/voices) from upstream vllm-project#1201 into the HT branch, adapted to coexist with HT's existing streaming and audio extraction changes. - Add upload_voice(), _load/_save_uploaded_speakers() to serving_speech - Add POST /v1/audio/voices endpoint to api_server - Modify GET /v1/audio/voices to include uploaded voice details - Auto-set ref_audio for uploaded voices in Base task - Add docs/serving/speech_api.md documentation Note: Known upstream review issues (path traversal, metadata locking, validation bypass for built-in voices) are carried as-is for parity and will be addressed in a follow-up. Upstream-PR: vllm-project#1201 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes security and logic issues flagged in upstream PR vllm-project#1201 review: Security: - Sanitize name/consent to alphanumeric/underscore/hyphen only - Validate resolved path stays within upload directory - Remove file_path from API responses (information disclosure) Logic bugs: - Base task validation now correctly requires ref_audio unless voice is specifically an uploaded voice (not just any voice name) - _get_uploaded_audio_data raises ValueError instead of returning None when audio file is missing, preventing silent failures Robustness: - Atomic metadata writes via tempfile + os.replace - File locking (fcntl.flock) on metadata.json reads and writes - Use Path().suffix for file extension extraction Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
I have made changes to these parts and added the speaker emb caching feature. |
|
fix precommits&DCO please |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| **Response Example:** | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "voice": { | ||
| "name": "custom_voice_1", | ||
| "consent": "user_consent_id", | ||
| "file_path": "/tmp/voice_samples/custom_voice_1_user_consent_id_1738660000.wav", | ||
| "created_at": 1738660000, | ||
| "mime_type": "audio/wav", | ||
| "file_size": 1024000 | ||
| } |
There was a problem hiding this comment.
The documented response example for POST /v1/audio/voices includes file_path, but the implementation intentionally does not return server file paths. Update the example to match the actual response schema (and also document the updated GET /v1/audio/voices response shape, which now includes uploaded_voices).
| @router.post( | ||
| "/v1/audio/voices", | ||
| responses={ | ||
| HTTPStatus.OK.value: {"model": dict}, | ||
| HTTPStatus.BAD_REQUEST.value: {"model": ErrorResponse}, | ||
| HTTPStatus.INTERNAL_SERVER_ERROR.value: {"model": ErrorResponse}, | ||
| }, | ||
| ) | ||
| async def upload_voice( | ||
| raw_request: Request, | ||
| audio_sample: UploadFile = File(...), | ||
| consent: str = Form(...), | ||
| name: str = Form(...), | ||
| ): | ||
| """Upload a new voice sample for voice cloning. | ||
|
|
||
| Uploads an audio file that can be used as a reference for voice cloning | ||
| in Base task TTS requests. The voice can then be referenced by name | ||
| in subsequent TTS requests. | ||
|
|
||
| Args: | ||
| audio_sample: Audio file (max 10MB) | ||
| consent: Consent recording ID | ||
| name: Name for the new voice | ||
| raw_request: Raw FastAPI request | ||
|
|
||
| Returns: | ||
| JSON response with voice information | ||
| """ | ||
| handler = Omnispeech(raw_request) | ||
| if handler is None: | ||
| return base(raw_request).create_error_response(message="The model does not support Speech API") | ||
|
|
||
| try: | ||
| # Validate required parameters | ||
| if not consent: | ||
| return base(raw_request).create_error_response(message="consent is required") | ||
| if not name: | ||
| return base(raw_request).create_error_response(message="name is required") | ||
|
|
||
| # Upload the voice | ||
| result = await handler.upload_voice(audio_sample, consent, name) | ||
|
|
||
| return JSONResponse(content={ | ||
| "success": True, | ||
| "voice": result | ||
| }) |
There was a problem hiding this comment.
There are existing unit tests for Speech serving, but this PR adds a new POST /v1/audio/voices endpoint and new behavior for Base requests using voice without ref_audio (uploaded voices). Add tests that cover: successful upload (metadata persisted), duplicate name handling, file size/type validation, and that a subsequent Base request with voice=<uploaded> works without ref_audio.
| @router.post( | ||
| "/v1/audio/voices", | ||
| responses={ | ||
| HTTPStatus.OK.value: {"model": dict}, | ||
| HTTPStatus.BAD_REQUEST.value: {"model": ErrorResponse}, | ||
| HTTPStatus.INTERNAL_SERVER_ERROR.value: {"model": ErrorResponse}, | ||
| }, | ||
| ) | ||
| async def upload_voice( | ||
| raw_request: Request, | ||
| audio_sample: UploadFile = File(...), | ||
| consent: str = Form(...), | ||
| name: str = Form(...), | ||
| ): | ||
| """Upload a new voice sample for voice cloning. | ||
|
|
||
| Uploads an audio file that can be used as a reference for voice cloning | ||
| in Base task TTS requests. The voice can then be referenced by name | ||
| in subsequent TTS requests. | ||
|
|
||
| Args: | ||
| audio_sample: Audio file (max 10MB) | ||
| consent: Consent recording ID | ||
| name: Name for the new voice | ||
| raw_request: Raw FastAPI request | ||
|
|
||
| Returns: | ||
| JSON response with voice information | ||
| """ | ||
| handler = Omnispeech(raw_request) | ||
| if handler is None: | ||
| return base(raw_request).create_error_response(message="The model does not support Speech API") | ||
|
|
||
| try: | ||
| # Validate required parameters | ||
| if not consent: | ||
| return base(raw_request).create_error_response(message="consent is required") | ||
| if not name: | ||
| return base(raw_request).create_error_response(message="name is required") | ||
|
|
||
| # Upload the voice | ||
| result = await handler.upload_voice(audio_sample, consent, name) | ||
|
|
||
| return JSONResponse(content={ | ||
| "success": True, | ||
| "voice": result | ||
| }) | ||
|
|
There was a problem hiding this comment.
POST /v1/audio/voices persists uploaded files under SPEECH_VOICE_SAMPLES with no retention/cleanup or quota beyond per-file size. Over time this can fill disk (especially with many uploads). Consider adding a configurable max number/total size of uploaded voices per server instance, and/or a delete endpoint / periodic cleanup policy.
| except Exception as e: | ||
| logger.error(f"Failed to save cache for speaker {speaker}: {e}") | ||
| # Update status to failed | ||
| self.update_metadata_cache_info(speaker, Path(""), "failed") | ||
| return False |
There was a problem hiding this comment.
In the exception path of save_voice_cache, update_metadata_cache_info is called with Path(""), which stringifies to "." and gets persisted as cache_file. This leaves misleading metadata and can point to a directory. Prefer clearing cache_file/cache_generated_at (e.g., write null/empty) when status is "failed", or update only the status field on failures.
|
|
||
| # Check cache status and file existence | ||
| if cache_file_path and cache_file_path.exists() and cache_status == "ready": | ||
| logger.info(f"Using cached voice clone prompt for speaker: {speaker}") | ||
|
|
||
| # Load cached VoiceClonePromptItem | ||
| cached_items = torch.load(cache_file_path, map_location="cpu", weights_only=False) |
There was a problem hiding this comment.
load_cached_voice_prompt loads cache_file from metadata.json and passes it directly into torch.load(..., weights_only=False). If metadata.json is tampered with, this can point outside the intended directory and torch.load can execute arbitrary code via pickle. Validate that cache_file_path resolves within SPEECH_VOICE_SAMPLES (and ideally only allow .pt files you created), and consider switching to a safer serialization format (e.g., store plain tensors in a JSON + safetensors/npz) so untrusted paths can’t trigger code execution.
| # Check cache status and file existence | |
| if cache_file_path and cache_file_path.exists() and cache_status == "ready": | |
| logger.info(f"Using cached voice clone prompt for speaker: {speaker}") | |
| # Load cached VoiceClonePromptItem | |
| cached_items = torch.load(cache_file_path, map_location="cpu", weights_only=False) | |
| # Validate cache file path before loading with torch.load | |
| resolved_cache_path: Optional[Path] = None | |
| if cache_file_path is not None: | |
| try: | |
| resolved_cache_path = cache_file_path.resolve() | |
| except Exception as exc: | |
| logger.warning( | |
| f"Failed to resolve cache file path for speaker {speaker}: {exc}" | |
| ) | |
| return None | |
| # If a base directory for speech voice samples is defined, ensure the | |
| # resolved cache path is within that directory to prevent directory traversal. | |
| base_dir = globals().get("SPEECH_VOICE_SAMPLES") | |
| if base_dir is not None: | |
| try: | |
| base_dir_path = Path(base_dir).resolve() | |
| resolved_cache_path.relative_to(base_dir_path) | |
| except Exception: | |
| logger.warning( | |
| f"Cache file path for speaker {speaker} is outside of the allowed " | |
| f"directory; refusing to load." | |
| ) | |
| return None | |
| # Optionally restrict to known-safe extension (e.g., .pt) | |
| if resolved_cache_path.suffix != ".pt": | |
| logger.warning( | |
| f"Unexpected cache file extension for speaker {speaker}: " | |
| f"{resolved_cache_path.suffix!r}; refusing to load." | |
| ) | |
| return None | |
| # Check cache status and file existence | |
| if resolved_cache_path and resolved_cache_path.exists() and cache_status == "ready": | |
| logger.info(f"Using cached voice clone prompt for speaker: {speaker}") | |
| # Load cached VoiceClonePromptItem | |
| cached_items = torch.load(resolved_cache_path, map_location="cpu", weights_only=False) |
| def _save_uploaded_speakers(self) -> None: | ||
| """Save uploaded speakers to metadata file.""" | ||
| try: | ||
| metadata = {"uploaded_speakers": self.uploaded_speakers} | ||
| with open(self.metadata_file, 'w') as f: | ||
| json.dump(metadata, f, indent=2) | ||
| except Exception as e: |
There was a problem hiding this comment.
_save_uploaded_speakers overwrites metadata.json with the in-memory self.uploaded_speakers dict. Since cache generation updates metadata.json from VoiceCacheManager independently, this write can silently clobber cache_status/cache_file/cache_generated_at (and concurrent uploads can lose updates/corrupt JSON). Use an atomic write + file lock, and merge/reload the on-disk metadata before writing so cache updates aren’t lost.
|
|
||
| # If voice is an uploaded speaker and no ref_audio provided, auto-set it | ||
| if request.voice.lower() in self.uploaded_speakers and request.ref_audio is None: | ||
| audio_data = self._get_uploaded_audio_data(request.voice) | ||
| if audio_data: | ||
| params["ref_audio"] = [audio_data] | ||
| params["x_vector_only_mode"] = [True] | ||
| logger.info(f"Auto-set ref_audio for uploaded voice: {request.voice}") | ||
| else: | ||
| raise ValueError(f"Audio file for uploaded voice '{request.voice}' is missing or corrupted") | ||
|
|
There was a problem hiding this comment.
For uploaded voices, _build_tts_params base64-encodes the audio file on every request when ref_audio isn’t provided, even though generate_voice_clone can use the on-disk cache after the first run. This adds avoidable CPU/memory overhead (up to 10MB read + encode per request). Consider checking cache_status == "ready" (and cache file existence) in metadata and skipping the ref_audio injection when the cache is already available.
| # If voice is an uploaded speaker and no ref_audio provided, auto-set it | |
| if request.voice.lower() in self.uploaded_speakers and request.ref_audio is None: | |
| audio_data = self._get_uploaded_audio_data(request.voice) | |
| if audio_data: | |
| params["ref_audio"] = [audio_data] | |
| params["x_vector_only_mode"] = [True] | |
| logger.info(f"Auto-set ref_audio for uploaded voice: {request.voice}") | |
| else: | |
| raise ValueError(f"Audio file for uploaded voice '{request.voice}' is missing or corrupted") | |
| # If voice is an uploaded speaker and no ref_audio provided, auto-set it | |
| if request.voice.lower() in self.uploaded_speakers and request.ref_audio is None: | |
| speaker_key = request.voice.lower() | |
| metadata = self.uploaded_speakers.get(speaker_key) | |
| # If a cache is already ready on disk for this uploaded voice, rely on it | |
| cache_status = None | |
| cache_path = None | |
| if isinstance(metadata, dict): | |
| cache_status = metadata.get("cache_status") | |
| cache_path = metadata.get("cache_path") | |
| if cache_status == "ready" and cache_path and os.path.exists(cache_path): | |
| # Cache is ready; no need to inject ref_audio (generate_voice_clone will use cache) | |
| logger.info( | |
| "Using existing cache for uploaded voice '%s'; skipping ref_audio injection", | |
| request.voice, | |
| ) | |
| else: | |
| audio_data = self._get_uploaded_audio_data(request.voice) | |
| if audio_data: | |
| params["ref_audio"] = [audio_data] | |
| params["x_vector_only_mode"] = [True] | |
| logger.info(f"Auto-set ref_audio for uploaded voice: {request.voice}") | |
| else: | |
| raise ValueError( | |
| f"Audio file for uploaded voice '{request.voice}' is missing or corrupted" | |
| ) |
zhaotyer
force-pushed
the
add_custom_voice
branch
4 times, most recently
from
e5140f1 to
b71ab32
Compare
|
PR #1206 merged. Please rebase |
zhaotyer
force-pushed
the
add_custom_voice
branch
from
b71ab32 to
d1ce0d8
Compare
|
pleas resolve conflicts |
🤖 Code Review: PR #1201 — Voice Upload API 🔴Verdict: Request Changes Summary: Adds REST endpoints to upload, list, and delete custom voice samples for voice cloning. Extends OmniOpenAIServingSpeech with uploaded_speakers management and auto-sets ref_audio + x_vector_only_mode when an uploaded voice is used. Architecture is clean with good separation between API layer (serving_speech.py) and storage/caching (VoiceSampleManager in qwen3_tts.py). This is the best-tested of the 4 PRs. However, there are path traversal vulnerabilities, state persistence issues on restart, and race conditions on concurrent uploads that must be fixed. Key Concerns:
Detailed Feedback📄
|
|
@vllm-omni-reviewer |
|
fix precommits please |
|
Please rebase on latest main. Also a couple issues:
|
linyueqian
force-pushed
the
add_custom_voice
branch
2 times, most recently
from
a5e6bba to
e161519
Compare
|
Rebased onto main, resolved conflicts, and fixed pre-commit issues. Also fixed the Tested locally with the Base model - upload, list, TTS with uploaded voice, duplicate rejection, and delete all work correctly. @hsliuustc0106 this should be ready to merge. |
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Tested locally, LGTM.
|
Uploaded voices always use |
|
Hi @zhaotyer , I appreciate that you addressed this issue which can occur in production environments. Can you please clarify the following:
|
Signed-off-by: linyueqian <linyueqian@outlook.com>
Signed-off-by: linyueqian <linyueqian@outlook.com>
…mment Signed-off-by: linyueqian <linyueqian@outlook.com>
Signed-off-by: linyueqian <linyueqian@outlook.com>
linyueqian
force-pushed
the
add_custom_voice
branch
from
10cfa28 to
52c3aa9
Compare
Signed-off-by: linyueqian <linyueqian@outlook.com> Co-authored-by: linyueqian <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com> Signed-off-by: lishunyang <lishunyang12@163.com>
|
|
||
| # Get uploaded speakers details | ||
| uploaded_speakers = [] | ||
| if hasattr(handler, "uploaded_speakers"): | ||
| for voice_name, info in handler.uploaded_speakers.items(): | ||
| uploaded_speakers.append( | ||
| { | ||
| "name": info.get("name", voice_name), | ||
| "consent": info.get("consent", ""), | ||
| "created_at": info.get("created_at", 0), | ||
| "file_size": info.get("file_size", 0), | ||
| "mime_type": info.get("mime_type", ""), | ||
| } | ||
| ) | ||
|
|
||
| return JSONResponse(content={"voices": speakers, "uploaded_voices": uploaded_speakers}) | ||
|
|
||
|
|
||
| @router.post( | ||
| "/v1/audio/voices", | ||
| responses={ |
There was a problem hiding this comment.
@Sy0307 @linyueqian hi, I have a quick question. Did this router get deleted?
There was a problem hiding this comment.
i think it may be deleted by accident. let me check if i can revert it.
There was a problem hiding this comment.
it's already back in main :-) @Sy0307
Yeah I got it. Just thanks for your guys' report and fix. I missed some emails last week XD
|
@zhaotyer Once a custom voice is added to the Base model, how to use it for TTS? Is there a python client code for this, which support streaming? |
Signed-off-by: linyueqian <linyueqian@outlook.com> Co-authored-by: linyueqian <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
The new endpoints allow users to:
Purpose
support add voice upload API for Qwen3-TTS
Test Plan
Test Result
Details
Files changed: - vllm_omni/entrypoints/openai/api_server.py - vllm_omni/entrypoints/openai/serving_speech.py - examples/online_serving/qwen3_tts/README.mdBEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)