Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions vllm/lora/layers/fused_moe.py
Original file line number Diff line number Diff line change
Expand Up @@ -196,14 +196,20 @@ def wrapper(*args, **kwargs):
config_dtype=config_dtype,
)

# SPARSITY_FACTOR is a heuristic margin ensuring tokens * top_k
# activates only a small fraction of total experts * loras.
SPARSITY_FACTOR = 8
naive_block_assignment = (
expert_map is None
and num_tokens * top_k * SPARSITY_FACTOR
<= self.base_layer.local_num_experts * self.max_loras
)
shrink_naive = shrink_config.pop("naive_block_assignment", None)
expand_naive = expand_config.pop("naive_block_assignment", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The keys in shrink_config and expand_config are normalized to uppercase by _normalize_keys (called within _get_lora_moe_configs at line 188). Consequently, popping the lowercase "naive_block_assignment" will always return None, causing the code to always fall back to the heuristic and ignoring any tuned values. You should use the normalized key "NAIVE_BLOCK_ASSIGNMENT" instead.

Suggested change
shrink_naive = shrink_config.pop("naive_block_assignment", None)
expand_naive = expand_config.pop("naive_block_assignment", None)
shrink_naive = shrink_config.pop("NAIVE_BLOCK_ASSIGNMENT", None)
expand_naive = expand_config.pop("NAIVE_BLOCK_ASSIGNMENT", None)


if shrink_naive is not None and expand_naive is not None:
# Both configs specify it — use True only if both True
naive_block_assignment = bool(shrink_naive and expand_naive)
else:
# Not in configs — fall back to heuristic
SPARSITY_FACTOR = 8
naive_block_assignment = (
expert_map is None
and num_tokens * top_k * SPARSITY_FACTOR
<= self.base_layer.local_num_experts * self.max_loras
)

# get the block size of m from customized config or default config
(
Expand Down Expand Up @@ -286,6 +292,8 @@ def wrapper(*args, **kwargs):
top_k=top_k,
config_dtype=config_dtype,
)
shrink_config.pop("naive_block_assignment", None)
expand_config.pop("naive_block_assignment", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similar to the issue in the activation decorator, the keys here are normalized to uppercase. Failing to pop the correct key means "NAIVE_BLOCK_ASSIGNMENT" remains in the configuration dictionaries, which may cause issues if passed down to underlying functions that do not expect it. Use the uppercase key name here as well.

Suggested change
shrink_config.pop("naive_block_assignment", None)
expand_config.pop("naive_block_assignment", None)
shrink_config.pop("NAIVE_BLOCK_ASSIGNMENT", None)
expand_config.pop("NAIVE_BLOCK_ASSIGNMENT", None)


sorted_token_ids_lora = moe_state_dict["sorted_token_ids_lora"]
expert_ids_lora = moe_state_dict["expert_ids_lora"]
Expand Down