From 80d9b098e1d8523c0a834841466a52f95a9dc66c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 12 Oct 2026 11:23:50 +0000 Subject: [PATCH 1/2] docs: update power-of-two-load-metric Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- .../concepts/routing/load-balancing.md | 22 +++++++++++++++++-- .../content/getting-started/load-balancing.md | 6 +++++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/src/lib/content/concepts/routing/load-balancing.md b/src/lib/content/concepts/routing/load-balancing.md index f13e642..c5c7602 100644 --- a/src/lib/content/concepts/routing/load-balancing.md +++ b/src/lib/content/concepts/routing/load-balancing.md @@ -184,7 +184,7 @@ Before it uses a request's bucket, the policy checks balance. The load it compar ## Power of Two Choices -Samples two distinct available workers at random and routes to the one with the lower expected wait, using the same formula as [`least_load`](#least-load): queued token work over throughput plus the KV-pressure penalty, with credit for requests sent since the last load report. If neither sampled worker has a load report, it compares their in-flight requests. It approaches least-load placement while reading only two workers' load per request. +Samples two distinct available workers at random and routes to the one with the lower expected wait, using the same formula as [`least_load`](#least-load): queued token work over throughput plus the KV-pressure penalty, with credit for requests sent since the last load report. If neither sampled worker has a load report, it compares their in-flight requests. It approaches least-load placement while reading only two workers' load per request. The expected wait is the default comparison; [`--power-of-two-load-metric`](#comparison-metric) can switch it to the gateway's own in-flight request counts. ```bash smg --policy power_of_two --worker-urls http://w1:8000 http://w2:8000 @@ -198,7 +198,7 @@ smg --policy power_of_two --worker-urls http://w1:8000 http://w2:8000 - Load-aware at two load reads per request - Counts queue depth, KV pressure and in-flight work -- No full-fleet scan per request +- No full-fleet scan per request (except the `least_requests` metric) @@ -214,6 +214,24 @@ smg --policy power_of_two --worker-urls http://w1:8000 http://w2:8000 +### Comparison Metric + +`--power-of-two-load-metric` sets what the policy compares its candidates by: + +| Value | Candidates | Comparison | +|-------|------------|------------| +| `expected_wait` | Two sampled at random | Lower expected wait wins (the default, described above) | +| `requests` | Two sampled at random | Fewer in-flight requests wins; the expected wait breaks a tie | +| `least_requests` | Every available worker | Keeps the workers with the fewest in-flight requests; the expected wait breaks a tie | + +```bash +smg --policy power_of_two --power-of-two-load-metric requests --worker-urls http://w1:8000 http://w2:8000 +``` + +Both in-flight metrics compare the gateway's own per-worker request counts (see [Load Signals](#load-signals)), read without any load report, so they balance on a live signal even where load reports are coarse or absent; only a tie goes on to the expected-wait score. `least_requests` gives up the policy's O(1) selection: it samples no pair and scans every available worker's count. Whatever the metric, a single available worker is selected directly. + +`--power-of-two-load-metric` is newer than v1.12.0. In v1.12.0 and earlier the comparison is always the expected wait. + **Use when:** Large or heterogeneous fleets where cache locality doesn't matter and each pick should read only two workers' load. --- diff --git a/src/lib/content/getting-started/load-balancing.md b/src/lib/content/getting-started/load-balancing.md index 752dade..235e7db 100644 --- a/src/lib/content/getting-started/load-balancing.md +++ b/src/lib/content/getting-started/load-balancing.md @@ -71,6 +71,12 @@ Samples two random workers and routes to the one with the lower expected wait, u smg --policy power_of_two --worker-urls http://w1:8000 http://w2:8000 ``` +| Parameter | Default | Description | +|-----------|---------|-------------| +| `--power-of-two-load-metric` | `expected_wait` | What candidates are compared by: `expected_wait` (lower expected wait of the sampled pair), `requests` (the pair's in-flight request counts, the expected wait breaking ties), or `least_requests` (no pair — the fewest in-flight requests across every available worker, then the expected wait) | + +`--power-of-two-load-metric` is newer than v1.12.0; in v1.12.0 and earlier the comparison is always the expected wait. The in-flight metrics compare the gateway's own request counts without reading load reports; see [Comparison Metric](../concepts/routing/load-balancing.md#comparison-metric). + Best for large or heterogeneous fleets where cache locality doesn't matter. --- From cf3ee0cbf9b23d0450736cabad3bc172131865a3 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Mon, 12 Oct 2026 14:07:52 +0000 Subject: [PATCH 2/2] docs: address feedback for #158 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- src/lib/content/concepts/routing/load-balancing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/lib/content/concepts/routing/load-balancing.md b/src/lib/content/concepts/routing/load-balancing.md index c5c7602..027867d 100644 --- a/src/lib/content/concepts/routing/load-balancing.md +++ b/src/lib/content/concepts/routing/load-balancing.md @@ -34,7 +34,7 @@ Routes to the worker with the lowest expected wait: queued token work over throu ### :material-scale-balance: Power of Two -Samples two workers and routes to the one with the lower expected wait. Near-least-load placement for two load reads per request. +Samples two workers and routes to the one with the lower expected wait; a flag can switch the comparison to in-flight request counts. Near-least-load placement for two load reads per request. @@ -81,7 +81,7 @@ Forwards every request to the single available worker without reading load or ca | `random` | :material-close: | :material-close: | :material-close: | O(1) | Testing | | `passthrough` | :material-close: | :material-close: | :material-close: | O(1) | Single worker | -Complexity is the selection cost per request over `n` available workers. [Sticky sessions](sticky-sessions.md) (`--routing-key-override`) add session affinity on top of any policy. +Complexity is the selection cost per request over `n` available workers; `power_of_two` becomes O(n) with [`--power-of-two-load-metric least_requests`](#comparison-metric), which scans every available worker. [Sticky sessions](sticky-sessions.md) (`--routing-key-override`) add session affinity on top of any policy. --- @@ -89,7 +89,7 @@ Complexity is the selection cost per request over `n` available workers. [Sticky Load-aware policies read one or both of two signals: -- **In-flight requests.** The gateway counts the requests it has sent to each worker that have not finished yet; a streamed response counts until its body ends. The count is kept for every request under every policy, on the HTTP, gRPC and PD paths alike, and is exported as `smg_worker_requests_active`. Each gateway replica counts only its own traffic. `prefix_hash` and the `cache_aware` spill gate compare these counts, and `least_load` and `power_of_two` fall back to them for workers without a load report. +- **In-flight requests.** The gateway counts the requests it has sent to each worker that have not finished yet; a streamed response counts until its body ends. The count is kept for every request under every policy, on the HTTP, gRPC and PD paths alike, and is exported as `smg_worker_requests_active`. Each gateway replica counts only its own traffic. `prefix_hash` and the `cache_aware` spill gate compare these counts, `least_load` and `power_of_two` fall back to them for workers without a load report, and [`--power-of-two-load-metric`](#comparison-metric) can make them `power_of_two`'s primary comparison. - **Load reports.** The gateway polls each worker's load every `--load-monitor-interval` seconds (default `10`). HTTP workers are asked for `/v1/loads` whatever the engine, falling back to the engine's Prometheus `/metrics` (SGLang, vLLM) when that route does not answer; gRPC and ZMQ workers report over their engine connection (`GetLoads`). A report carries the worker's waiting requests and KV-cache usage, plus queued tokens and generation throughput where the engine exposes them. `least_load`, `power_of_two` and `cache_aware` read these reports. See [Load Monitoring](../../reference/configuration.md#load-monitoring-configuration) for the polling options. In `least_load`, `power_of_two` and `cache_aware`, scores that tie exactly are broken uniformly at random, so equally idle workers share traffic instead of the first worker taking every tie.