Skip to content

[sgl-router] refactor - SLO ordering for bucket selection - #40292

Merged
ShangmingCai merged 1 commit into
mainfrom
codex/policies-reorg-slo
Sep 22, 2026
Merged

ShangmingCai merged 1 commit into
mainfrom
codex/policies-reorg-slo

Conversation

@sherlockwu

@sherlockwu sherlockwu commented Sep 19, 2026 •

Copy link
Copy Markdown
Collaborator

Add optional TTFT and token-throughput preferences to bucket selection. Length constraints remain mandatory; SLO preferences order the compatible buckets before engine selection. Each PD attempt still obtains both engines from one bucket, and nonpreferred buckets remain available for fallback.

Based directly on main; the cache-aware changes from #40366 and admission changes from #40271 are already merged.

Changes

  • Add optional service estimates to Bucket and independent resolver preferences: Disabled (default), SloFirst (matching first), or BestEffort (nonmatching first).
  • Weight unmet TTFT and throughput preferences equally, then break ties by capacity, rank, and ID. With both set to SloFirst, buckets matching both targets precede those matching one, followed by those matching neither.
  • Treat absent targets as neutral and missing or invalid estimates as nonmatching. Preserve input/context limits and admission fallback across SLO tiers.
  • Pass targets directly to BucketResolver::resolve. Parse x-sgl-ttft-slo-ms and x-sgl-tps-slo in the shared chat route only when their preference is enabled; invalid enabled headers return 400 before dispatch, while disabled headers are ignored.
  • Cover conflicting targets, mixed preferences, missing estimates, invalid inputs, length constraints, complete P/D pairing, and rejection fallback.

Preferences are configured programmatically on reorg resolvers. Legacy routing remains the default; production configuration construction remains a follow-up.

Validation after rebase

Rebased only the SLO commit onto main (9eda772a21), excluding the already-merged cache-aware and admission commits. Preserve BucketResolver::new stage validation and its Result return type from #40366; adapt SLO test constructors to that API. Cache-aware and admission policy implementations and their regression tests remain identical to main.

  • cargo fmt --all --check
  • cargo test --locked --workspace --lib --tests --quiet: 1,117 tests passed, none skipped.
  • cargo clippy --locked --workspace --all-targets -- -D warnings
  • git diff --check

CI States

Latest PR Test (Base): ✅ Run #35678290678
Latest PR Test (Extra): ❌ Run #35678290559
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 19, 2026
@sherlockwu
sherlockwu added this pull request to stack #40280 September 19, 2026 04:32
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from a8a0afb to ad42136 Compare September 19, 2026 04:42
@sherlockwu
sherlockwu removed this pull request from stack #40280 September 19, 2026 17:34
@sherlockwu
sherlockwu added this pull request to stack #40368 September 19, 2026 17:34
@sherlockwu
sherlockwu removed this pull request from stack #40368 September 19, 2026 20:40
@sherlockwu
sherlockwu added this pull request to stack #40380 September 19, 2026 20:40
@sherlockwu sherlockwu changed the title [sgl-router] Add optional SLO ordering for bucket selection [sgl-router] refactor - SLO ordering for bucket selection Sep 19, 2026
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch 2 times, most recently from 228bf18 to d62a3ae Compare September 20, 2026 17:28
@sherlockwu
sherlockwu removed this pull request from stack #40380 September 20, 2026 17:28
@sherlockwu
sherlockwu added this pull request to stack #40484 September 20, 2026 17:28
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from d62a3ae to f743139 Compare September 20, 2026 22:23
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from f743139 to 3991ff3 Compare September 20, 2026 23:49
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from 3991ff3 to bc9f462 Compare September 20, 2026 23:57
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from bc9f462 to 06b128d Compare September 21, 2026 00:22
Base automatically changed from router/policies-reorg-2-load to main September 21, 2026 08:26
@ShangmingCai
ShangmingCai marked this pull request as ready for review September 21, 2026 09:10

@ShangmingCai ShangmingCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic is correct and genuinely well-tested, after a restack/rebase, we could launch the CI and merge it after CI passes.

Comment on lines 241 to 246
pub fn new(buckets: Vec<Bucket>) -> Self {
Self { buckets }
Self {
buckets,
..Self::default()
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be careful with this part when restacking, since #40366 and #40271 are already in main

@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-slo branch from 06b128d to f9f5a00 Compare September 22, 2026 02:07
@ShangmingCai
ShangmingCai merged commit 59a723e into main Sep 22, 2026
93 of 97 checks passed
@ShangmingCai
ShangmingCai deleted the codex/policies-reorg-slo branch September 22, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants