Repository navigation
[Bugfix][MiniCPM-o] Apply the Talker's 16-frame codec penalty on MRv2 - #8576
Conversation
Since the vLLM 0.31 rebase the MRv2 sampler runs its penalty stage from `sampler.logits_processors` (`Sampler.add_request` registers requests with every list entry and `apply_sampling_params` calls each entry's `apply(logits, ctx)`). `_install_mrv2_talker_sampler` only rebound `sampler.penalties_state`, so the list kept the stock `PenaltiesState`: the Talker's codec window penalty never ran, and the stock presence penalty (repetition_penalty 1.05 over the whole prompt and output, including the placeholder prompt id 0) was applied instead. Make `_CodecWindowPenaltiesState` a vLLM 0.31 `LogitsProcessor` (`add_request` returns whether it applies, `apply(logits, ctx)`), put it in the stock penalty state's slot of the sampler's own pipeline, and keep `penalties_state` pointing at it for the runner's output bin counts. Add a CUDA test that drives the real MRv2 `Sampler` and checks the processed logits match the V1 `_apply_batched_repetition_penalty`. Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
|
This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md. Module owners: @tzhouam @fake0fan @Gaohan123 Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files @linyueqian, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Omni ReviewBot routing recordAssigned Strict on cursor (cursor-grok-4.6-high) under experiment |
hsliuustc0106
added
the
high priority
ChatGPT Codex Connector
Sy0307
left a comment
Sy0307
left a comment
There was a problem hiding this comment.
The replacement reaches the real vLLM 0.31 sampler and restores the 16-frame codec repetition penalty while preserving frequency and presence penalties. H200 validation passed 1,139 tests (5 skipped); the new regression fails with the base implementation, and independent real-sampler checks matched the oracle through mixed penalties, row reordering, and slot reuse. No blocking issue found; this validation did not include full-model generation or long-form audio quality.
Reuse the existing Talker prompt-window recipe for native MRv2 updates, retain the codec penalty window across conditions, and flush confirmed sampled EOS without ending a live session early. Integrates the sampler pipeline registration fix from vllm-project#8576. Add V1/MRv2 live-client CI coverage and remove independent H200 precision tuning from this PR. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…vllm-project#8576) Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
4 participants
Purpose
On the MRv2 runner, the MiniCPM-o 4.5 Talker is meant to apply a repetition penalty only over the codec codes of the last 16 frames, as the V1 path does in
_apply_batched_repetition_penalty. Since the vLLM 0.31 rebase (#8459) that penalty never runs.In vLLM 0.31 the MRv2
Samplerbuildsself.logits_processors = [LogitBiasState, self.penalties_state, BadWordsState, *custom]in__init__, registers every request with each entry, and applies them throughprocessor.apply(logits, ctx)._install_mrv2_talker_sampleronly reassignedsampler.penalties_state, so the list kept the stockPenaltiesState: the stock 1.05 penalty ran over the whole prompt and output (including the dummy prompt id 0), and the Talker's window penalty was never called. The wrapper also still used the 0.30 interface.This change, scoped to the MiniCPM-o Talker:
_CodecWindowPenaltiesStateimplements vLLM 0.31'sLogitsProcessor:add_requestreturns whether processing is needed, andapply(logits, ctx)applies the 16-frame window repetition penalty over output codes, then delegates frequency/presence to the stock state with its repetition factor neutralized._install_mrv2_talker_samplerreplaces the stock penalty entry in that sampler instance'slogits_processorsby identity (and fails loudly if it is missing), and keepspenalties_statepointing at the wrapper so the runner still readsoutput_bin_counts. Nothing is patched globally; other MRv2 samplers are unchanged.custom_logits_processorscannot disable the stock penalty, which is why the entry is replaced rather than appended.Test Plan
vLLM Version: 0.31.0
vLLM-Omni Commit: this branch on
d5a338010test_mrv2_sampler_applies_codec_window_penalty_instead_of_stock_penaltydrives the real 0.31Sampler/RequestStateand compares logits with V1's_apply_batched_repetition_penalty.pytest tests/model_executor/models/minicpmo_4_5 tests/worker_v2on H200.turn_mrv2profile, 128 Seed-TTS EN prompts at C=8, main vs this branch, plus the default V1 profile; Whisper large-v3. Both MRv2 arms carry [Core][Model] Unified Full-Duplex MRv2 Support and MiniCPM-o 4.5 Adaptation #8536's one-lineomni_ar_model_runner.pyfix locally (not part of this PR) so the separate duplicated-hidden-state bug does not mask this change.Test Result
The new test fails on main (3 of 32 logits differ: prompt id 0, an out-of-window code, and an in-window code penalized once instead of 1.05^3) and passes with this change. The GPU suites pass: 1161 passed, 5 skipped.
Processor list on the Talker sampler: main
['LogitBiasState', 'PenaltiesState', 'BadWordsState'], this PR['LogitBiasState', '_CodecWindowPenaltiesState', 'BadWordsState']. While serving, the logits changed per row equal the distinct codes in the last 16 frames, id 0 is never touched, and factors are 1/1.05 and 1/1.05^2 by frequency, as on V1.The two items above 0.3 are one-word ASR slips. Throughput is unchanged within noise. Seed-TTS EN clips are about 4 s, so this run cannot show the long-answer drift the window penalty targets; no long-form A/B was run.