Skip to content

Separate allocation logic from scheduler #11313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cctry merged 22 commits into from
Oct 11, 2025
Merged

Separate allocation logic from scheduler #11313

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
9f238a4
match_prefix
cctry Oct 2, 2025
c0fc6a0
fix
cctry Oct 3, 2025
baa34d7
prepare_for_extend: adjust order
cctry Oct 3, 2025
ffc57c0
prepare_for_extend: unify indices writing
cctry Oct 3, 2025
75e3cf7
clean
cctry Oct 3, 2025
3fe0dd6
clean
cctry Oct 3, 2025
ebb29fc
set prefix_indices default
cctry Oct 3, 2025
5fa6bbf
fix chunk cache
cctry Oct 4, 2025
9d7b32f
prepare_for_extend
cctry Oct 6, 2025
a30f56b
move functions out
cctry Oct 7, 2025
4d4a272
better
cctry Oct 7, 2025
e3e1d28
wip
cctry Oct 7, 2025
217fa5b
wip
cctry Oct 8, 2025
2b54ecc
Merge remote-tracking branch 'origin/main' into shiyang/mem_v2/alloc
cctry Oct 8, 2025
ba41c12
clean
cctry Oct 8, 2025
d498464
clean
cctry Oct 8, 2025
4e0bb42
clean
cctry Oct 8, 2025
6c29a88
Merge remote-tracking branch 'origin/main' into shiyang/mem_v2/alloc
cctry Oct 8, 2025
2e8d1a4
fix benchmark
cctry Oct 8, 2025
987a3dd
fix hybrid
cctry Oct 8, 2025
bd22a18
update error msg
cctry Oct 10, 2025
563ba3a
swa
cctry Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion python/sglang/bench_one_batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
import multiprocessing
import os
import time
from types import SimpleNamespace
from typing import Tuple

import numpy as np
Expand Down Expand Up @@ -257,11 +258,18 @@ def prepare_synthetic_inputs_for_latency_test(

@torch.no_grad
def extend(reqs, model_runner):
# Create dummy tree_cache for benchmarks (no prefix caching, just allocation)
dummy_tree_cache = SimpleNamespace(
page_size=1,
Copy link
Copy Markdown
Collaborator

@airMeng airMeng Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why hard-code to 1 instead page_size=model_runner.server_args.page_size ?

Copy link
Copy Markdown
Collaborator Author

@cctry cctry Oct 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out. wIll fix this

device=model_runner.device,
token_to_kv_pool_allocator=model_runner.token_to_kv_pool_allocator,
)

batch = ScheduleBatch.init_new(
reqs=reqs,
req_to_token_pool=model_runner.req_to_token_pool,
token_to_kv_pool_allocator=model_runner.token_to_kv_pool_allocator,
tree_cache=None,
tree_cache=dummy_tree_cache,
model_config=model_runner.model_config,
enable_overlap=False,
spec_algorithm=SpeculativeAlgorithm.NONE,
Expand Down
Loading
Loading