Skip to content

[sgl-router] refactor - cache-aware policy - #40366

Merged
ShangmingCai merged 2 commits into
mainfrom
codex/policies-reorg-cache-aware
Sep 21, 2026
Merged

[sgl-router] refactor - cache-aware policy#40366
ShangmingCai merged 2 commits into
mainfrom
codex/policies-reorg-cache-aware

Conversation

@sherlockwu

@sherlockwu sherlockwu commented Sep 19, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Add cache-aware selection to the bucket engine groups introduced by #40241. Plain and prefill groups can prefer workers holding a usable prefix, while cache misses fall back to power-of-two selection within that group's soft queue tier.

Stack: main → #40272 (shared state) → #40241 (bucket routing and power-of-two) → #40379 (session-aware) → #40366 (cache-aware) → #40271 (admission) → #40292 (SLO ordering).

  • Support the shared local radix tree and remote prefix indexer, preserving worker hashing, exact-URL matching, unavailable-backend fallback, and fatal query rejection.
  • Memoize prefix lookups once per prepared request and shared source. Bucket retries reuse I/O but rerun candidate filtering, pressure checks, and admission. Separate index sources retain separate memo entries.
  • Preserve minimum token/ratio thresholds, bounded candidates, the switch margin, pressure guard, soft queue diversion, and saturation pinning. Use bounded tournaments rather than sorting with a non-transitive pressure comparator.
  • Check cache candidates and the fallback winner through EngineAdmission. Hard rejection advances to another bucket; saturation never bypasses admission. Concrete named metric limits are integrated in [sgl-router] refactor - generalized admission policy definitions #40271.

The HTTP cache regression seeds the prefix index with tokens produced by the configured tokenizer, matching request preparation on this base.

Legacy routing remains the default. Reorg policies are constructed programmatically; production config wiring, power-of-k fallback configuration, and cache decision metrics remain follow-ups.

Validation after rebase

Rebased the stack onto main (95521da18d) after #39867 merged. All 31 commit patches and each PR's router tree are unchanged. Formatting and diff checks pass after this rebase. Clippy and workspace tests passed on the identical router tree in the previous rebase: 1,100 passed, with two Hugging Face cache-dependent parity tests excluded.

PR-specific validation from the unchanged router trees:

  • cargo +nightly fmt --all -- --check
  • cargo +1.92 clippy --locked --workspace --all-targets -- -D warnings
  • cargo +1.92 test --locked --test component --test proxy -- --skip parity_matrix: 103 component and 136 proxy tests passed. Two Hugging Face cache-dependent parity tests excluded.
  • Cache-aware policy changes replay unchanged on [sgl-router] refactor - session-aware policy #40379; cache/indexer component tests and the tokenized HTTP routing regression pass.

CI States

Latest PR Test (Base): ✅ Run #35553896652
Latest PR Test (Extra): ❌ Run #35553896587
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 19, 2026
This was referenced Sep 19, 2026
Merged
Merged
@sherlockwu
sherlockwu added this pull request to stack #40368 September 19, 2026 17:34
This was referenced Sep 19, 2026
Merged
Merged
@sherlockwu
sherlockwu marked this pull request as draft September 19, 2026 20:23
@sherlockwu sherlockwu mentioned this pull request Sep 19, 2026
Merged
@sherlockwu
sherlockwu removed this pull request from stack #40368 September 19, 2026 20:40
@sherlockwu
sherlockwu changed the base branch from codex/policies-reorg-skeleton to codex/policies-reorg-session-aware September 19, 2026 20:40
@sherlockwu
sherlockwu added this pull request to stack #40380 September 19, 2026 20:40
@sherlockwu sherlockwu changed the title [sgl-router] refactor - cache-aware engine selection [sgl-router] refactor - cache-aware policy Sep 19, 2026
sherlockwu added a commit that referenced this pull request Sep 19, 2026
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
0277756 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 236b3b7 to 0277756 Compare September 19, 2026 23:09
sherlockwu added a commit that referenced this pull request Sep 19, 2026
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
f455093 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 0277756 to f455093 Compare September 19, 2026 23:15
@sherlockwu
sherlockwu marked this pull request as ready for review September 20, 2026 00:02
sherlockwu added a commit that referenced this pull request Sep 20, 2026
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
386f656 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from f455093 to 386f656 Compare September 20, 2026 00:04
sherlockwu added a commit that referenced this pull request Sep 20, 2026
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
c2de8bb 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 386f656 to c2de8bb Compare September 20, 2026 17:28
@sherlockwu
sherlockwu removed this pull request from stack #40380 September 20, 2026 17:28
@sherlockwu
sherlockwu added this pull request to stack #40484 September 20, 2026 17:28
sherlockwu added a commit that referenced this pull request Sep 20, 2026
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
7281f5c 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from c2de8bb to 7281f5c Compare September 20, 2026 22:23
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 7281f5c to 8aa10f1 Compare September 20, 2026 23:49
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 8aa10f1 to 1802ef2 Compare September 20, 2026 23:57
Base automatically changed from codex/policies-reorg-session-aware to main September 21, 2026 00:22
@sherlockwu @claude
[sgl-router] Add cache-aware selection to bucket engine groups
0ce1985 
CacheAwarePolicy picks within one plain or prefill group following the
legacy cache_aware proposal and resolve_cache_candidates rules: exact-URL
prefix holders from a local radix tree or remote indexer, hit thresholds,
bounded candidates, soft queue gate, saturation pin, pressure guard, and
EngineAdmission on every candidate and on the power-of-two fallback pick.
PrefixMemo shares one lookup per request and source across bucket attempts.

Squashed from #40366 and rebased onto the renamed load-monitor modules
(engine_reported_load, router_inflight_load).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sherlockwu
sherlockwu force-pushed the codex/policies-reorg-cache-aware branch from 1802ef2 to 0ce1985 Compare September 21, 2026 00:22
@sherlockwu @claude
[sgl-router] Reject stage-incompatible bucket policies when the resol…
cdc7cc7 
…ver is built

A CacheAwarePolicy installed on a decode group was only detected inside
pick(), where the chat reorg path treats InvalidConfiguration as fatal and
turns every request into a 500. Policies now declare the stages they
support and BucketResolver::new validates each bucket's groups up front.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ShangmingCai
ShangmingCai merged commit fcb080b into main Sep 21, 2026
92 of 97 checks passed
@ShangmingCai
ShangmingCai deleted the codex/policies-reorg-cache-aware branch September 21, 2026 05:53
@ShangmingCai ShangmingCai mentioned this pull request Sep 21, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ShangmingCai ShangmingCai ShangmingCai approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sherlockwu @ShangmingCai