Repository navigation
[sgl-router] refactor - move policy-required states under src/state - #40272
Merged
Merged
Conversation
sherlockwu
force-pushed
the
router/policies-reorg-0-state-move
branch
from
September 19, 2026 01:06
399af99 to
85c6702
Compare
sherlockwu
added this pull request to stack #40280
September 19, 2026 01:09
This was referenced Sep 19, 2026
sherlockwu
changed the title
sherlockwu
removed this pull request from stack #40280
September 19, 2026 17:34
sherlockwu
added this pull request to stack #40368
September 19, 2026 17:34
sherlockwu
changed the title
sherlockwu
removed this pull request from stack #40368
September 19, 2026 20:40
sherlockwu
added this pull request to stack #40380
September 19, 2026 20:40
sherlockwu
marked this pull request as ready for review
September 19, 2026 22:49
sherlockwu
force-pushed
the
router/policies-reorg-0-state-move
branch
from
September 19, 2026 22:53
dfe320f to
a889213
Compare
sherlockwu
changed the title
sherlockwu
force-pushed
the
router/policies-reorg-0-state-move
branch
from
September 20, 2026 00:04
a889213 to
5ccb03e
Compare
An error occurred while trying to automatically change base from
codex/refactor-router-config
to
main
September 20, 2026 11:40
ShangmingCai
approved these changes
Sep 20, 2026
ShangmingCai
left a comment
ShangmingCai
left a comment
Collaborator
There was a problem hiding this comment.
Safe and correct, only need a rebase.
sherlockwu
force-pushed
the
router/policies-reorg-0-state-move
branch
from
September 20, 2026 17:28
5ccb03e to
f8aed5b
Compare
sherlockwu
requested review from
Fridge003,
Ying1123,
hzh0425,
ispobock,
merrymercy and
ping1jing2
as code owners
September 20, 2026 17:28
sherlockwu
requested review from
AgainstEntropy,
JustinTong0323,
Kangyan-Zhou,
bingxche,
ishandhanani,
wisclmy0611,
yctseng0211,
yichiche and
zijiexia
as code owners
September 20, 2026 17:28
github-actions
Bot
added
documentation
sherlockwu
removed this pull request from stack #40380
September 20, 2026 17:28
sherlockwu
changed the base branch from
codex/refactor-router-config
to
main
September 20, 2026 17:28
sherlockwu
added this pull request to stack #40484
September 20, 2026 17:28
Pure relocation: kv_events/ becomes state/kv_events/, and engine_load.rs
and active_load.rs become state/load_monitor/{engine_load,active_load}.rs.
Every import path follows; no code changes. This gives the legacy policies
and the upcoming policies_reorg one home for the signals they both read.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…policies Both policies kept the same private state: a map of key to assigned worker with a last-seen instant, a clock, an idle window and a sweep. That state becomes state::AffinityStore, keyed by WorkerId, and both policies hold an Arc to it. Two small semantic tightenings: sticky pins by worker id rather than URL, and a racing first touch for one key converges on the binding that is still a candidate instead of last-writer-wins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rename engine-published measurements to engine_reported_load and router-local request accounting to router_inflight_load. Make AffinityStore::sweep_expired return the number of entries it removed instead of a length delta, which under-counted when requests inserted keys during the sweep. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sherlockwu
force-pushed
the
router/policies-reorg-0-state-move
branch
from
September 20, 2026 22:23
f8aed5b to
dbb8cd6
Compare
3 of 5 tasks
ighutake-debug
added a commit
to ighutake-debug/sglang
that referenced
this pull request
Sep 28, 2026
…ection path Fixes sgl-project#32752: the counter was defined, rendered, documented, and unit tested, but no production route ever called record_decode_affinity, so live PD deployments never reported decode-affinity outcomes. - select_decode_with_affinity now returns the branch taken alongside the worker (same_host_picked / fallback_breaker / fallback_load_imbalance / new fallback_no_same_host for pools with no co-located peer) - LegacyHostAffinityDecodePolicy tags its SelectionProposal with the outcome; select_decode_peer records it once per successful selection via a metrics handle on DecodeSelectionInputs (None in tests, the shared registry in the chat route) - a capacity-fallback override of the affinity pick records fallback_load_imbalance, since the affinity was not honored - plain-mode requests and failed selections record nothing, per the issue's expected behavior Rebuilt on current main after the sgl-router state/bucket refactors (sgl-project#40241, sgl-project#40272) — the previous branch auto-merged cleanly but produced unparsable code in the merge tree, which is what lint caught.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move the routing state required by policies to make routing decisions into src/state/. This includes load monitoring, KV events, and affinity stores such as session state.
Stack: main → #40272 (shared state) → #40241 (bucket routing and power-of-two) → #40379 (session-aware) → #40366 (cache-aware) → #40271 (admission) → #40292 (SLO ordering).
Changes
state::kv_eventsand load monitoring tostate::load_monitor.engine_reported_load) from router-local request accounting (router_inflight_load).AffinityStore. Sticky routing reuses eligible bindings during concurrent first touches; session-aware routing retains its final-selection commit behavior.WorkerIdinstead of URL. Static discovery uses the URL as the ID, so nothing changes there; on Kubernetes the ID is the pod UID, so a replaced pod at the same address is remapped instead of reused, since the new pod has no warm cache.Validation after rebase
Rebased the stack onto
main(95521da18d) after #39867 merged. All 31 commit patches and each PR's router tree are unchanged. Formatting and diff checks pass after this rebase. Clippy and workspace tests passed on the identical router tree in the previous rebase: 1,100 passed, with two Hugging Face cache-dependent parity tests excluded.PR-specific validation from the unchanged router trees:
cargo +nightly fmt --all -- --checkcargo +1.92 clippy --locked --workspace --all-targets -- -D warningscargo +1.92 test --locked --workspace -- --skip parity_matrix: 1,037 tests passed. Two Hugging Face cache-dependent parity tests excluded.CI States
Latest PR Test (Base): ✅ Run #35541557061
Latest PR Test (Extra): ❌ Run #35541556949
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.