Skip to content

[BugFix] Fix the issue of incorrect generated tokens - #4461

Merged
Gaohan123 merged 1 commit into
vllm-project:mainfrom
amy-why-3459:bugfix
Jun 17, 2026
Merged

Gaohan123 merged 1 commit into
vllm-project:mainfrom
amy-why-3459:bugfix

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Jun 16, 2026 •

Copy link
Copy Markdown
Collaborator

PLEASE FILL IN THE PR DESCRIPTION HERE.

Purpose

Fix #4459

Test Plan

vLLM Version: v0.23.0

vLLM-Omni Commit: c58a02c

vllm bench serve \
    --omni \
  --dataset-name random \
  --port 28889 \
  --max-concurrency 1 \
  --model /data/why/Qwen3-Omni-30B-A3B-Instruct \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --num-prompts 1 \
  --random-input-len 2500 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf \
  --random-output-len 900 \
  --extra_body '{"modalities": ["text"]}'

Test Result

============ Serving Benchmark Result ============
Successful requests:                     1         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  4.32      
Request throughput (req/s):              0.23      
Peak concurrent requests:                1.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          4318.05   
Median E2EL (ms):                        4318.05   
P99 E2EL (ms):                           4318.05   
================== Text Result ===================
Total input tokens:                      2508      
Total generated tokens:                  900       
Output token throughput (tok/s):         208.40    
Peak output token throughput (tok/s):    230.00    
Peak concurrent requests:                1.00      
Total Token throughput (tok/s):          789.14    
---------------Time to First Token----------------
Mean TTFT (ms):                          410.83    
Median TTFT (ms):                        410.83    
P99 TTFT (ms):                           410.83    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.35      
Median TPOT (ms):                        4.35      
P99 TPOT (ms):                           4.35      
---------------Inter-token Latency----------------
Mean ITL (ms):                           4.35      
Median ITL (ms):                         3.26      
P99 ITL (ms):                            7.54      
==================================================
image

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix looks correct. Using continuous_usage_stats to resolve token deltas per SSE chunk and expanding ITL accordingly properly accounts for bundled tokens.

@Gaohan123 Gaohan123 added this to the v0.24.0 milestone Jun 17, 2026

@Gaohan123 Gaohan123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks

@Gaohan123 Gaohan123 added the ready label to trigger buildkite CI label Jun 17, 2026
@Gaohan123
Gaohan123 enabled auto-merge (squash) June 17, 2026 02:05
@Gaohan123
Gaohan123 merged commit d897852 into vllm-project:main Jun 17, 2026
6 of 8 checks passed
syd520zy pushed a commit to syd520zy/vllm-omni that referenced this pull request Jun 17, 2026
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: syd520zy <529477025@qq.com>
syd520zy pushed a commit to syd520zy/vllm-omni that referenced this pull request Jun 17, 2026
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: syd520zy <529477025@qq.com>
Nughm3 pushed a commit to Nughm3/vllm-omni that referenced this pull request Jun 18, 2026
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: [Benchmark] The number of generated tokens is incorrect.

3 participants