Repository navigation
Enable Tunable Control for naive_block_assignment #34920
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
cf2bfa8
6742e4f
161f963
715797b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -196,14 +196,20 @@ def wrapper(*args, **kwargs): | |||||||||
| config_dtype=config_dtype, | ||||||||||
| ) | ||||||||||
|
|
||||||||||
| # SPARSITY_FACTOR is a heuristic margin ensuring tokens * top_k | ||||||||||
| # activates only a small fraction of total experts * loras. | ||||||||||
| SPARSITY_FACTOR = 8 | ||||||||||
| naive_block_assignment = ( | ||||||||||
| expert_map is None | ||||||||||
| and num_tokens * top_k * SPARSITY_FACTOR | ||||||||||
| <= self.base_layer.local_num_experts * self.max_loras | ||||||||||
| ) | ||||||||||
| shrink_naive = shrink_config.pop("naive_block_assignment", None) | ||||||||||
| expand_naive = expand_config.pop("naive_block_assignment", None) | ||||||||||
|
|
||||||||||
| if shrink_naive is not None and expand_naive is not None: | ||||||||||
| # Both configs specify it — use True only if both True | ||||||||||
| naive_block_assignment = bool(shrink_naive and expand_naive) | ||||||||||
| else: | ||||||||||
| # Not in configs — fall back to heuristic | ||||||||||
| SPARSITY_FACTOR = 8 | ||||||||||
| naive_block_assignment = ( | ||||||||||
| expert_map is None | ||||||||||
| and num_tokens * top_k * SPARSITY_FACTOR | ||||||||||
| <= self.base_layer.local_num_experts * self.max_loras | ||||||||||
| ) | ||||||||||
|
|
||||||||||
| # get the block size of m from customized config or default config | ||||||||||
| ( | ||||||||||
|
|
@@ -286,6 +292,8 @@ def wrapper(*args, **kwargs): | |||||||||
| top_k=top_k, | ||||||||||
| config_dtype=config_dtype, | ||||||||||
| ) | ||||||||||
| shrink_config.pop("naive_block_assignment", None) | ||||||||||
| expand_config.pop("naive_block_assignment", None) | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similar to the issue in the activation decorator, the keys here are normalized to uppercase. Failing to pop the correct key means
Suggested change
|
||||||||||
|
|
||||||||||
| sorted_token_ids_lora = moe_state_dict["sorted_token_ids_lora"] | ||||||||||
| expert_ids_lora = moe_state_dict["expert_ids_lora"] | ||||||||||
|
|
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The keys in
shrink_configandexpand_configare normalized to uppercase by_normalize_keys(called within_get_lora_moe_configsat line 188). Consequently, popping the lowercase"naive_block_assignment"will always returnNone, causing the code to always fall back to the heuristic and ignoring any tuned values. You should use the normalized key"NAIVE_BLOCK_ASSIGNMENT"instead.