Repository navigation
feat(router): conditional disaggregation #11357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
a948b8b
2e4735e
1d170ab
0b7b469
e68488c
f022c6b
f8d7836
48bbf30
8add3b5
d064f42
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,8 +12,10 @@ | |
| AicPerfConfigBase, | ||
| ) | ||
| from dynamo.common.configuration.groups.kv_router_args import ( | ||
| CONDITIONAL_DISAGG_POLICY_CHOICES, | ||
| KvRouterArgGroup, | ||
| KvRouterConfigBase, | ||
| warn_conditional_disagg_prefill_busy_threshold_resolution, | ||
| ) | ||
| from dynamo.common.configuration.utils import add_argument, add_negatable_bool_argument | ||
| from dynamo.llm import AicPerfConfig, KvRouterConfig | ||
|
|
@@ -66,6 +68,27 @@ def validate(self) -> None: | |
| "--router-prefill-load-model=aic requires " | ||
| "--router-track-prefill-tokens" | ||
| ) | ||
| if self.conditional_disagg_policy not in CONDITIONAL_DISAGG_POLICY_CHOICES: | ||
| raise ValueError( | ||
| "--router-conditional-disagg-policy must be one of " | ||
| + ", ".join( | ||
| f"'{choice}'" for choice in CONDITIONAL_DISAGG_POLICY_CHOICES | ||
| ) | ||
| ) | ||
| if self.conditional_disagg_eff_isl_threshold < 0: | ||
| raise ValueError( | ||
| "--router-conditional-disagg-eff-isl-threshold must be >= 0" | ||
| ) | ||
| if not 0.0 <= self.conditional_disagg_eff_isl_ratio_threshold <= 1.0: | ||
| raise ValueError( | ||
| "--router-conditional-disagg-eff-isl-ratio-threshold must be in [0.0, 1.0]" | ||
| ) | ||
| if self.conditional_disagg_enabled: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The standalone router accepts and forwards this flag, but its handler constructs only
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed all conditional disagg related code for standalone router. Looks like standalone router doesn't acknowledge the kv router flags it does not support (rather than reject those flags explicitly)? |
||
| warn_conditional_disagg_prefill_busy_threshold_resolution( | ||
| policy=self.conditional_disagg_policy, | ||
| busy_threshold=self.conditional_disagg_prefill_busy_threshold, | ||
| queue_threshold=self.router_queue_threshold, | ||
| ) | ||
|
|
||
|
|
||
| class DynamoRouterArgGroup(ArgGroup): | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.