Skip to content

[Bugfix] out-of-bounds error for routed experts capture #37118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HollowMan6 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Bugfix] out-of-bounds error for routed experts capture #37118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions vllm/v1/core/sched/scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,16 +263,12 @@ def __init__(
if isinstance(group.kv_cache_spec, AttentionSpec):
self.routed_experts_attn_gid = gid
break
min_block_size = min(
[
group.kv_cache_spec.block_size
for group in kv_cache_config.kv_cache_groups
]
)
num_groups = len(kv_cache_config.kv_cache_groups)
attn_group = kv_cache_config.kv_cache_groups[self.routed_experts_attn_gid]
# Routed experts are read back with the attention group's slot_mapping,
# so the shared-memory view must match that group's full KV address space.
self.max_num_kv_tokens = (
kv_cache_config.num_blocks // num_groups
) * min_block_size
kv_cache_config.num_blocks * attn_group.kv_cache_spec.block_size
)
Comment thread
HollowMan6 marked this conversation as resolved.
Comment thread
HollowMan6 marked this conversation as resolved.
dcp_size = self.vllm_config.parallel_config.decode_context_parallel_size
pcp_size = self.vllm_config.parallel_config.prefill_context_parallel_size
if pcp_size * dcp_size > 1:
Expand Down
14 changes: 5 additions & 9 deletions vllm/v1/worker/gpu_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6543,16 +6543,12 @@ def init_routed_experts_capturer(self):
)
routed_experts_capturer = RoutedExpertsCapturer.create()
self.routed_experts_attn_gid = self._get_attention_kv_cache_gid()
min_block_size = min(
[
group.kv_cache_spec.block_size
for group in self.kv_cache_config.kv_cache_groups
]
)
num_groups = len(self.kv_cache_config.kv_cache_groups)
attn_group = self.kv_cache_config.kv_cache_groups[self.routed_experts_attn_gid]
# Routed experts are indexed with the attention group's slot_mapping,
# so the side buffer must match that group's full KV address space.
self.max_num_kv_tokens = (
self.kv_cache_config.num_blocks // num_groups
) * min_block_size
self.kv_cache_config.num_blocks * attn_group.kv_cache_spec.block_size
)
Comment thread
HollowMan6 marked this conversation as resolved.
Comment thread
HollowMan6 marked this conversation as resolved.
dcp_size = self.vllm_config.parallel_config.decode_context_parallel_size
pcp_size = self.vllm_config.parallel_config.prefill_context_parallel_size
if pcp_size * dcp_size > 1:
Expand Down
Loading