[Router] Add bounded-load consistent hashing policy

[waizuichougou]

Summary

Adds an opt-in bounded_consistent_hashing policy for SGLang Model Gateway routing-key affinity.

Closes #33625

Motivation

Strict consistent_hashing can keep a hot routing key on an overloaded worker while other healthy workers are underutilized. This change adds bounded-load spillover without changing the existing strict policy contract.

Modifications

• Add bounded_consistent_hashing with configurable max_load_skew, defaulting to 1.5.
• Preserve the preferred consistent-hash worker while its load is within the healthy-worker average bound.
• Walk the existing hash ring clockwise for the first healthy eligible worker when the preferred worker is overloaded.
• Keep X-SMG-Target-Worker, implicit affinity headers, and the existing consistent_hashing policy strict.
• Add bounded-policy branch metrics and hold WorkerLoadGuard across regular HTTP streaming and non-streaming request lifetimes.
• Add CLI/config validation, documentation, and unit coverage.
• Expose the policy and max_load_skew through the Python launcher and PyO3 bindings.

Accuracy Tests

Not applicable; this change affects gateway worker selection and does not change model outputs.

Speed Tests and Profiling

Not applicable; this change adds a routing policy and does not modify model execution or kernel code.

Test Plan

• cargo fmt --all -- --check — passed.
• cargo check --all-targets — passed.
• cargo test --lib — 399 passed.
• cargo test --lib policies::bounded_consistent_hashing -- --nocapture — 7 passed.
• pytest tests/test_arg_parser.py tests/test_router_config.py tests/test_pyo3_binding.py tests/test_validation.py -q — 158 passed, 3 skipped.
• cargo check --manifest-path bindings/python/Cargo.toml — passed.
• cargo test --test load_guard_raii_test — 6 passed.
• cargo test --lib policies::factory::tests — 2 passed.
• git diff --check — passed.

Checklist

• Format the code with rustfmt.
• Add unit tests.
• Update the gateway documentation.
• Keep the existing consistent_hashing behavior unchanged.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ✅ Run #31067350734
Latest PR Test (Extra): ❌ Run #31067350643

[waizuichougou]

/tag-and-rerun-ci

[chengcuiping]

Hi @waizuichougou — thanks for implementing #33625.

A production user has now provided relevant feedback in #33625: the ratio-only guard can spill a routing key at near-idle because the preferred worker’s initial in-flight increments dominate the very small fleet average.

The current branch’s load_ok() uses:

worker_load <= max(avg_load, 1.0) * max_load_skew

so rebasing alone would retain that edge case. The proposed correction is to require both a relative overload and a minimum absolute load gap before spillover, together with a regression covering an idle fleet followed by concurrent requests sharing one routing key.

Would you be open to reopening and rebasing this PR? If you plan to continue carrying it, I would be glad to help with the threshold correction and regression coverage. If not, please let me know whether you are comfortable with me building on the existing commits with clear attribution. I will avoid opening a duplicate implementation in the meantime.

[waizuichougou]

Hi @chengcuiping

Thanks for the detailed feedback. I don't plan to continue carrying this PR, so please feel free to build on the existing commits and take over the implementation.

Please use your judgment regarding attribution. If the follow-up implementation makes substantial use of the existing code, I would appreciate being credited as a co-author where appropriate.

I’m happy for you to open a follow-up PR and continue the fix. Thanks for helping move this forward.

[chengcuiping]

Thanks @waizuichougou — understood. I’ll take this forward as a new follow-up PR based on the current main.

I’ll preserve the authorship of the existing commits and credit both you and #33791 in the PR body. If the history later needs to be squashed, I’ll use the exact author identity from the original commits for the Co-authored-by attribution rather than guessing it.

The follow-up will retain the existing policy boundary, replace the ratio-only near-idle guard with an absolute-plus-relative condition, and add focused near-idle and concurrent-routing-key regressions before the HTTP A/B validation.

I’ll link the new PR here once it is ready. Thanks for making the handoff clear.