Skip to content

[BugFix] process request.num_cached_tokens if it equals to the initial value #1468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hsliuustc0106 merged 7 commits into from
Feb 26, 2026
+4 −1
Merged

[BugFix] process request.num_cached_tokens if it equals to the initial value #1468

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f634edf
fix bug
LJH-LBJ Feb 25, 2026
7b76d92
fix bug
LJH-LBJ Feb 25, 2026
7afb173
rerun
LJH-LBJ Feb 25, 2026
83fb4b5
Merge branch 'main' into fix-num_cached_tokens-not-process
LJH-LBJ Feb 25, 2026
c627f18
Merge branch 'main' into fix-num_cached_tokens-not-process
Gaohan123 Feb 25, 2026
8a37dd0
increase Omni Model Test timeout_in_minutes
LJH-LBJ Feb 26, 2026
da6858e
Merge branch 'main' into fix-num_cached_tokens-not-process
LJH-LBJ Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/test-ready.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ steps:
# type: DirectoryOrCreate

- label: "Omni Model Test"
timeout_in_minutes: 15
timeout_in_minutes: 17
depends_on: upload-ready-pipeline
commands:
- export VLLM_LOGGING_LEVEL=DEBUG
Expand Down
3 changes: 3 additions & 0 deletions vllm_omni/core/sched/omni_generation_scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@ def schedule(self) -> SchedulerOutput:
# Pop the finished request from waiting queue and don't schedule it
self.waiting.pop_request()
continue
# Count the number of prefix cached tokens.
if request.num_cached_tokens < 0:
request.num_cached_tokens = request.num_computed_tokens

# async_chunk: wait for the first upstream chunk (don't start with placeholders).
if self.chunk_transfer_adapter is not None and len(request.prompt_token_ids) == 0:
Expand Down