[Model] Support HyperCLOVAX-SEED-Omni-8B#585
Conversation
effortprogrammer
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| elif self.model_stage == "code2wav" and self.code2wav is not None: | ||
| # Input to Code2Wav stage are the Audio Tokens generated by Thinker | ||
| audio_tokens = input_ids | ||
| waveform = self.code2wav(audio_tokens) | ||
| return OmniOutput( |
There was a problem hiding this comment.
The code2wav forward path returns an OmniOutput (lines 149‑153) but the class never sets have_multimodal_outputs. GPUGenerationModelRunner calls OmniGPUModelRunner.extract_multimodal_outputs (vllm_omni/worker/gpu_model_runner.py:314-331), which only accepts OmniOutput when the model advertises have_multimodal_outputs; otherwise it raises ValueError("Invalid hidden states type"). As soon as the code2wav stage runs via the generation worker, this missing flag causes a hard crash before any audio is produced.
Useful? React with 👍 / 👎.
|
@effortprogrammer Hey, the HyperCLOVAX-SEED-Omni-8B draft doesn't have a test plan or results yet. Were you able to run inference with the Naver checkpoint? I see #869 and #613 are also working on the audio and vision decoder parts — are you coordinating with those PRs? |
|
@lishunyang12 I apologize for the delayed update. I heard from a Naver employee (the company behind this model) that there will be an update regarding the OMNI model. They mentioned that they plan to submit a new PR to support the OMNI model in the vllm-omni repo after the update. I will close this PR now. |
2 participants
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
Purpose
Add new model: https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Omni-8B
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)