Skip to content

[Bugfix][Higgs-Audio-V3] Propagate Sampling Parameters to Sampler - #5406

Merged
Gaohan123 merged 5 commits into
vllm-project:mainfrom
sphinxkkkbc:bugfix/fix_serving_speech_sampling_metadata
Jul 25, 2026
Merged

Gaohan123 merged 5 commits into
vllm-project:mainfrom
sphinxkkkbc:bugfix/fix_serving_speech_sampling_metadata

Conversation

@sphinxkkkbc

@sphinxkkkbc sphinxkkkbc commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

PLEASE FILL IN THE PR DESCRIPTION HERE.

Purpose

Fix #5387

Enable per-request sampling parameters for Higgs Audio v3 speech generation.

  • Promotes temperature, top_p, and top_k from speech request extra_params into the stage-0 SamplingParams.
  • Allows vLLM to propagate these values through InputBatch into SamplingMetadata.
  • Passes SamplingMetadata to the Higgs Audio v3 codebook sampler.
  • Expands request-level sampling parameters from [batch] to [batch * num_vq], so every VQ logits row uses the parameters of its owning request.

Test Plan

  • Validate speech request sampling parameters are propagated through stage-0 SamplingParams and SamplingMetadata to the model-owned sample() method.
  • Validate Higgs Audio v3 correctly applies per-request temperature, top_p, and top_k across batched requests and their VQ logits, top_k=-1 and invalid_temperature test
pytest -v tests/unit/higgs_audio_v3/test_higgs_audio_v3.py
pytest -v tests/worker/test_gpu_ar_model_runner.py 

vLLM Version:
0.25.0
vLLM-Omni Commit:
b57b76a

Test Result

  • test_gpu_ar_model_runner: 21 passed
  • test_higgs_audio_v3: 77 passed

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@sphinxkkkbc sphinxkkkbc changed the title [BugFix][Higgs-Audio-V3]Remove Hardcoded Sampling_Params and accept sampling_metadata in online serving [Bugfix][Higgs-Audio-V3] Propagate Sampling Parameters to the Sampler Jul 25, 2026
@sphinxkkkbc sphinxkkkbc changed the title [Bugfix][Higgs-Audio-V3] Propagate Sampling Parameters to the Sampler [Bugfix][Higgs-Audio-V3] Propagate Sampling Parameters to Sampler Jul 25, 2026
@Gaohan123 Gaohan123 added this to the v0.26.0 milestone Jul 25, 2026
mask.scatter_(-1, sorted_idx, sorted_mask)
x = x.masked_fill(mask, float("-inf"))
temperatures, top_ks, top_ps = self._expand_audio_sampling_params(sampling_metadata, x, num_codebooks)
top_ks = top_ks.clamp(min=1, max=x.shape[-1])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

top_k=-1 is commonly used by vLLM callers to mean “disable top-k filtering,” but clamp(min=1, ...) converts it to 1. That makes audio-code sampling greedy for those requests, rather than leaving the distribution unrestricted.

Could we preserve the no-top-k sentinel before calling apply_top_k_top_p—for example, map top_k <= 0 to x.shape[-1] (or the helper’s documented disable value)? Please also add a regression test covering top_k=-1, ensuring more than one candidate remains sampleable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, test added and descriptions updated

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
…com:sphinxkkkbc/vllm-omni into bugfix/fix_serving_speech_sampling_metadata
@Gaohan123 Gaohan123 added the ready label to trigger buildkite CI label Jul 25, 2026

@Gaohan123 Gaohan123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks

@Gaohan123
Gaohan123 enabled auto-merge (squash) July 25, 2026 15:06
@Gaohan123
Gaohan123 merged commit 76e4a78 into vllm-project:main Jul 25, 2026
6 of 9 checks passed
nagisa-kunhah pushed a commit to nagisa-kunhah/vllm-omni that referenced this pull request Jul 30, 2026
…lm-project#5406)

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
…lm-project#5406)

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Higgs Audio v3 (TTS) Sampling Parameters ignored in sampling of audio codes

2 participants