Repository navigation
Conversation
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
| mask.scatter_(-1, sorted_idx, sorted_mask) | ||
| x = x.masked_fill(mask, float("-inf")) | ||
| temperatures, top_ks, top_ps = self._expand_audio_sampling_params(sampling_metadata, x, num_codebooks) | ||
| top_ks = top_ks.clamp(min=1, max=x.shape[-1]) |
There was a problem hiding this comment.
top_k=-1 is commonly used by vLLM callers to mean “disable top-k filtering,” but clamp(min=1, ...) converts it to 1. That makes audio-code sampling greedy for those requests, rather than leaving the distribution unrestricted.
Could we preserve the no-top-k sentinel before calling apply_top_k_top_p—for example, map top_k <= 0 to x.shape[-1] (or the helper’s documented disable value)? Please also add a regression test covering top_k=-1, ensuring more than one candidate remains sampleable.
There was a problem hiding this comment.
fixed, test added and descriptions updated
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
…com:sphinxkkkbc/vllm-omni into bugfix/fix_serving_speech_sampling_metadata
…lm-project#5406) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
…lm-project#5406) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
PLEASE FILL IN THE PR DESCRIPTION HERE.
Purpose
Fix #5387
Enable per-request sampling parameters for Higgs Audio v3 speech generation.
temperature,top_p, andtop_kfrom speech requestextra_paramsinto the stage-0SamplingParams.InputBatchintoSamplingMetadata.SamplingMetadatato the Higgs Audio v3 codebook sampler.[batch]to[batch * num_vq], so every VQ logits row uses the parameters of its owning request.Test Plan
SamplingParamsandSamplingMetadatato the model-ownedsample()method.temperature,top_p, andtop_kacross batched requests and their VQ logits, top_k=-1 and invalid_temperature testvLLM Version:
0.25.0
vLLM-Omni Commit:
b57b76a
Test Result
BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)