Repository navigation
[sgl-router] refactor - generalized admission policy definitions - #40271
Merged
Merged
Conversation
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 19, 2026 00:35
35d4008 to
85dae09
Compare
This was referenced Sep 19, 2026
sherlockwu
changed the title
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 19, 2026 01:08
85dae09 to
4c349b2
Compare
sherlockwu
added this pull request to stack #40280
September 19, 2026 01:09
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 19, 2026 04:29
4c349b2 to
fcb5845
Compare
github-actions
Bot
added
the
documentation
sherlockwu
changed the title
sherlockwu
changed the title
sherlockwu
removed this pull request from stack #40280
September 19, 2026 17:34
sherlockwu
changed the base branch from
codex/policies-reorg-skeleton
to
codex/policies-reorg-cache-aware
September 19, 2026 17:34
sherlockwu
added this pull request to stack #40368
September 19, 2026 17:34
sherlockwu
removed this pull request from stack #40368
September 19, 2026 20:40
sherlockwu
added this pull request to stack #40380
September 19, 2026 20:40
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 19, 2026 23:29
07fbd4f to
37acb03
Compare
sherlockwu
changed the title
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 19, 2026 23:51
37acb03 to
6d27855
Compare
sherlockwu
marked this pull request as ready for review
September 19, 2026 23:55
sherlockwu
changed the title
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 20, 2026 00:05
6d27855 to
225c94e
Compare
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 20, 2026 17:28
225c94e to
1e0b6ec
Compare
sherlockwu
removed this pull request from stack #40380
September 20, 2026 17:28
sherlockwu
added this pull request to stack #40484
September 20, 2026 17:28
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 20, 2026 22:23
1e0b6ec to
092d76e
Compare
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 20, 2026 23:49
092d76e to
6cb9f56
Compare
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 20, 2026 23:57
6cb9f56 to
0a10b5c
Compare
sherlockwu
force-pushed
the
router/policies-reorg-2-load
branch
from
September 21, 2026 00:22
0a10b5c to
53d5a01
Compare
An admission policy is a set of per-engine caps. `AdmissionLimits` holds one optional cap per metric (running, waiting, KV tokens, pending prefill tokens, router-local in-flight); unset caps are not checked and the default admits everything. Policies read the selected engine's `EngineMetrics` from the selection snapshot plus the live in-flight counter and admit while every capped metric is below its cap. Request size stays a bucket concern; unknown reported metrics fail open; the rejection reason is the cap's name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ShangmingCai
force-pushed
the
router/policies-reorg-2-load
branch
from
September 21, 2026 05:54
53d5a01 to
8deda3e
Compare
ShangmingCai
added a commit
to ShangmingCai/sglang
that referenced
this pull request
Sep 21, 2026
The previous commit renamed `AllowAll` but also reworded the prose around
it ("default limits", "unless limits are configured"), editing text sgl-project#40271
wrote instead of aligning to it. It also spelled the concept two different
ways.
Use the name sgl-project#40271 established itself, `AdmissionLimits::default()`, in
all four places and restore every other word. Normalizing whitespace on
sgl-project#40271's POLICY_DESIGN.md and applying that one rename now reproduces this
file exactly; lines are rewrapped only where the longer name no longer
fits the paragraph.
No behavior change; documentation and one comment only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ShangmingCai
approved these changes
Sep 21, 2026
ShangmingCai
left a comment
ShangmingCai
left a comment
Collaborator
There was a problem hiding this comment.
Looks good. Will fix some minor issues (test compile, doc) in #40537
Collaborator
|
/tag-and-rerun-ci |
ShangmingCai
added a commit
to ShangmingCai/sglang
that referenced
this pull request
Sep 21, 2026
sgl-project#40271 was squash-merged as a987101, so this branch's copy of its commit (8deda3e) is not in main's history and every file they both touch conflicts textually despite identical content. Resolve by taking the tree of this branch rebased onto main: the sgl-project#40271 content comes from main's squash commit, plus this PR's two changes (the BucketResolver::new unwrap and the AllowAll rename). All three files are byte-identical to the previously verified branch tip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: main → #40272 (shared state) → #40241 (bucket routing and power-of-two) → #40379 (session-aware) → #40366 (cache-aware) → #40271 (admission) → #40292 (SLO ordering).
Design
AdmissionLimitshas one optional cap per metric:max_running_requests,max_waiting_requests,max_kv_tokens,max_pending_prefill_tokens,max_inflight_requests. An unset cap is not checked; the default admits everything. Serde-shaped (deny_unknown_fields) so CLI/config can map to it 1:1.EngineMetrics::observe(engine, snapshot)reads the selected engine's measurements from the load snapshot selection already captured, plus the live router in-flight counter. Reported metrics areNonewithout a fresh, complete report, never zero.EngineAdmission::check(engine, &metrics)admits while every capped metric is below its cap and rejects with the cap's name. Unknown reported metrics fail open. Request size is not an admission input: buckets already select by input length and context capacity, and admission observes load without reserving it.admission: Arc<dyn EngineAdmission>and decide where to check (power-of-two on its pick, session-aware on binding reuse/assignment, cache-aware per candidate and on its fallback winner). Rejection advances to the next bucket as before.Production CLI/config construction of resolvers and their admission limits remains separate work.
Validation after rebase
Rebased the stack onto
main(b3e4d198af) after #39867 merged. All 31 commit patches and each PR's router tree are unchanged. The rebased stack tip (#40292) passes formatting, Clippy, and workspace tests: 1,100 passed, with two Hugging Face cache-dependent parity tests excluded.PR-specific validation from the unchanged router trees:
cargo +nightly fmt --all -- --checkcargo +1.92 clippy --locked --workspace --all-targets -- -D warningscargo +1.92 test --locked --test component --test proxy -- --skip parity_matrix: 106 component and 137 proxy tests passed. Two Hugging Face cache-dependent parity tests excluded.CI States
Latest PR Test (Base): ✅ Run #35566292682
Latest PR Test (Extra): ❌ Run #35566292519
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.