[Router] Add bounded consistent hashing with an absolute load gap

[chengcuiping]

Recommended review order:

1. Core policy and spill condition
2. HTTP active-load lifetime
3. Config and Python binding
4. Tests and documentation

Fixes #33625
Supersedes #33791

This follow-up continues #33791 with permission from @waizuichougou.
The original implementation commits are preserved with their authorship.

Permission:
#33791 (comment)

Summary

• Keep bounded_consistent_hashing as an independent opt-in policy; strict
  consistent_hashing is completely unchanged.
• Add the policy-specific configurable min_load_gap, measured in active
  requests.
• Allow bounded spill only for an explicit X-SMG-Routing-Key, and only when
  both the absolute and relative load thresholds are exceeded.
• Keep X-SMG-Target-Worker and implicit keys from Authorization,
  X-Forwarded-For, and Cookie strict.
• In non-IGW mode, route over a cached aggregate ring containing both Regular
  and HTTP workers.
• Rebuild that ring only when membership changes; the request hot path does not
  construct a ring.
• Retain the preferred worker when no eligible spill candidate qualifies.
• Track active load for the full streaming and non-streaming request lifecycle.

Why existing policies do not cover this case

This policy addresses a narrower contract than the existing alternatives:

• manual with assignment_mode=min_load assigns a new routing key to the
  least-loaded worker and then stores that assignment in gateway-local mutable
  state. Separate gateway replicas cannot independently derive the same
  assignment without shared state.
• prefix_hash hashes token prefixes and applies a ratio-only load factor. The
  regular Model Gateway HTTP path does not supply tokens, and its affinity key
  is not X-SMG-Routing-Key.
• cache_aware detects fleet-wide max/min imbalance and switches to
  shortest-queue routing. It does not preserve deterministic routing-key
  placement or ring-order failover.
• PR dp-attention: add prefix_affinity load balancing for routing-key/session affinity #31170 applies bounded affinity among native intra-instance DP ranks. It cannot select among independent backend workers behind Model Gateway.
• Strict consistent_hashing remains unchanged for deployments where affinity
  is a correctness requirement rather than a cache-locality preference.

Threshold rationale and limitations

The two threshold terms intentionally use different references:

• preferred_load - min_healthy_load > min_load_gap requires that moving the request can reduce active load by a meaningful absolute amount.
• preferred_load > mean_healthy_load * max_load_skew requires that the preferred worker is an outlier relative to the healthy fleet. Using the
  minimum as the ratio denominator would be unstable whenever an idle worker has load zero.

The parameter names are kept distinct from cache-aware
balance_abs_threshold / balance_rel_threshold because the compared workers, selection behavior, and policy scope differ. Naming can be adjusted if the Model Gateway maintainers prefer a shared configuration vocabulary.

Limitations. The load signal is local to each gateway process, so multiple gateway replicas may make different best-effort spill decisions. A spill also sacrifices worker-local KV locality and may require a cold prefill on the selected worker. This policy is therefore intended for cache-locality optimization only and must not be used when backend-local session state makes affinity a correctness requirement.

Spill condition

preferred_load.saturating_sub(min_healthy_load) > min_load_gap
    && preferred_load > mean_healthy_load * max_load_skew

Both comparisons are strict.

Two-worker HTTP A/B and default status

The two-worker HTTP A/B comprised 5 repetitions, 20 cells, 400 groups, and
3,200 streaming requests.

Policy	Throughput vs strict	Mean TTFT	Placement	Spill
strict	baseline	3087.04 ms	8:0	N/A
gap=1	+2.330%, CI [1.016110, 1.035140]	2684.51 ms	6:2	25%
gap=2	+2.376%, CI [1.016512, 1.035208]	2688.84 ms	6:2	25%
gap=4	+2.099%, CI [1.012597, 1.031751]	2694.72 ms	6:2	25%

• All 3,200/3,200 requests returned HTTP 200 with matching output and no
  counter leakage.
• Bounded routing changed the frozen eight-sibling placement from 8:0 to 6:2
  and reduced mean TTFT by about 13%.
• The cached-token fraction fell from 75.546% under strict routing to about
  56.95%, while prefill compute increased by about 76%.
• This explicitly trades cache reuse and prefill work for better load balance
  and latency.
• Direct paired confidence intervals among gaps 1, 2, and 4 could not establish
  that any of the three settings differs from either of the others; all three
  converged to the same 25% spill and 6:2 placement.
• The benchmark supports the bounded mechanism, but it does not support a claim
  that min_load_gap=2 is optimal. The value 2 remains only a conservative,
  configurable middle default for this opt-in policy.

The GPU A/B ran at SHA
10a3b95f510584174d8b8c661c7974b97d72291a. The subsequent commits
75b73b0c5adc8c92b648caf984cd0cb28ab42877 and 356299712d1bc649efed655dc610838096b80d3a contain only
documentation/comment clarification and Black formatting; neither changes
runtime semantics.

Production signal and limitation

hassellof reported successful affinity and a 3/2/1/2 placement during a storm
drill:

#33625 (comment)

That drill did not record any diverts. It therefore did not exercise the
gateway spill branch and is not a complete validation of spillover behavior.

Validation

• cargo fmt --all -- --check
• cargo check --locked --all-targets
• Bounded-policy Rust tests: 18 passed
• Full Rust library: 414 passed
• Load guard: 8 passed
• Production Router→HTTP regression: 5 independent runs passed
• Python:
  • 31 passed, 1 expected skip
  • 21 passed, 1 expected skip
  • 79 passed
• Exact-SHA GPU A/B: 3,200/3,200 passed
• Lint, Router, SMG benchmark, and current Base/platform CI passed
• Any Draft-gated Extra workflow state reflects workflow gating or permissions,
  not a code failure
• git diff --check
• git diff --check origin/main...HEAD

---

CI States

Latest PR Test (Base): ✅ Run #33489527894
Latest PR Test (Extra): ❌ Run #33489527729
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

[chengcuiping]

@slin1237, the planned two-worker HTTP A/B is complete and the PR is now ready for review.

Across 3,200 streaming requests, bounded routing changed the frozen 8-sibling placement from 8:0 to 6:2, reduced mean TTFT by about 13%, and improved throughput by 2.10–2.38%, with correctness passing for every request.

The experiment did not distinguish min_load_gap=1, 2, and 4: all three converged to the same 25% spill and final placement. I therefore retain 2 only as a conservative, configurable middle default for this opt-in policy, not as an empirically optimal value. The PR description also discloses the corresponding cache-hit and prefill-compute trade-off.

The non-IGW aggregate-ring correction and real Router→HTTP regression coverage are included as well. I would appreciate feedback on the policy contract and configuration surface.

[chengcuiping]

@slin1237 @ByronHsu, could you please confirm whether the separate bounded_consistent_hashing policy and its configuration surface are acceptable for review? The branch is mergeable, and Base, Lint, Router, and SMG checks are green; Extra is only gated on the CI label. I’m happy to adjust or split the scope if you prefer a different direction.

[chengcuiping]

Hi @slin1237 @ByronHsu, before I do another current-main refresh, could you please give a direction-level decision on whether SMG wants a separate opt-in bounded_consistent_hashing policy?

The implementation and A/B validation are complete, and strict consistent_hashing remains unchanged. If the policy direction is acceptable, I’ll refresh the branch and address review feedback. If this policy surface is not desired, please let me know and I’m happy to close the PR while retaining the benchmark findings in #33625. If scope is the concern, I can also split it in the way maintainers prefer.