docs: power_of_two can compare candidates by in-flight requests: the --power-of-two-load-metric setting

[github-actions]

What this PR does

What does the power_of_two policy compare its candidates by — is the least-load expected wait the only metric, or can --power-of-two-load-metric switch it to the router's own in-flight request counts (requests, least_requests), and what exactly does each value do?

Why we need it

Source change: smg-project/smg@3f12f12

Commit 21 (#3092, 3f12f121, 2026-10-11); its supplied diff was read and every claim verified on the current default-branch snapshot. crates: model_gateway/src/config/types.rs:917-935 PowerOfTwoLoadMetric { expected_wait (default), requests, least_requests }; model_gateway/src/main.rs:488-489 defines --power-of-two-load-metric (default expected_wait; parse_power_of_two_load_metric at :202-204 rejects other values naming the three spellings), wired at :1808; the Python launcher matches (bindings/python/src/smg/router_args.py:350 and :638-642, choices expected_wait/requests/least_requests). Semantics (model_gateway/src/policies/power_of_two.rs:51-84): expected_wait — two distinct healthy workers sampled at random, the lower least-load expected wait wins (unchanged default); requests — the same random pair is first compared by the router's own in-flight request count (fewest_in_flight :40-49, worker.load(), each count read once), and the expected wait breaks a tie; least_requests — no random pair: every healthy worker is filtered to the fewest in-flight requests and the expected wait breaks the tie, a full-fleet scan. One healthy worker short-circuits in every mode, and exact ties still break uniformly at random inside the expected-wait scorer. The in-flight modes read no load reports for the first comparison, so they work where load reports are coarse or absent; the expected-wait tiebreak still uses least_load's default tuning. Incomplete docs, re-read tonight: concepts/routing/load-balancing.md:185-217 (Power of Two Choices) — ':187 Samples two distinct available workers at random and routes to the one with the lower expected wait ... If neither sampled worker has a load report, it compares their in-flight requests' presents expected wait as the only metric and in-flight counts as only a no-report fallback, and the section never mentions the new setting; getting-started/load-balancing.md:66-74 — ':68 Samples two random workers and routes to the one with the lower expected wait' — same gap. What stays: ':211 Always uses least_load's default tuning; the --least-load-* flags do not apply' remains true (the expected-wait scorer is shared, untouched). Boundary: reference/configuration.md:89's policy row and the new flag's reference row are the deploy-config scan's (and configuration.md is consumed by my admission item tonight); the behavior contract lives on the two load-balancing pages, which are this concern's canonical homes. The same-area #3091 (commit 19, skip the power_of_two worker-count startup check under --dp-aware) is a separate startup-validation concern and is not bundled. Version framing: #3092 (2026-10-11) postdates the v1.12.0 bump (commit 299, #2888); present the setting as current-main behavior newer than v1.12.0 — in v1.12.0 and earlier the comparison was always the expected wait. One independent concern: one commit made one policy's comparison metric selectable, documented in the two sections that describe that policy. Dedup: no open or closed PR mentions #3092 or power-of-two-load-metric; open PRs #144-150 hold neither file; my other items touch neither (the consistent_hashing Authorization correction, #2899, also lives on these pages and is deferred in remaining_work, not bundled). Edit well under 1,000 lines.

Scope: load-balancing / power-of-two-load-metric. Other concerns are deferred.

How to test

• Passed the documentation path and size guard (under 1000 added plus deleted lines; no file-count limit).
• Passed an independent accuracy and single-concern review.
• Passed git diff --check and pnpm check and the production pnpm build.

Checklist

• Tests added/updated (if applicable)
• Docs updated (if applicable)
• Commit includes a DCO sign-off

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 29f84a05-883b-46fd-82d2-492ac37764e6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Documentation maintenance: ready. Unsuccessful content rounds: 0/3. Operational attempts: 0/3.

Head: 8489c4445b1e813a9577804af86ad03e96a3d809; reviewed main: 8df4f5b4b0429633d2a22f92c67a2c93640477d5.
Reviewed SMG source: a29c64d4323dbf21950e2822cb099af04f6fd901.

Every claim verified against the SMG checkout at a29c64d4: --power-of-two-load-metric exists with default expected_wait and values expected_wait/requests/least_requests on both the Rust CLI (main.rs:503, parser :212-215) and Python launcher (router_args.py:661-666), and applies to PD leg policies via the shared parse_policy (main.rs:2377-2394). Semantics match power_of_two.rs:51-84: single available worker short-circuits in every mode; requests compares the sampled pair's router in-flight counts (fewest_in_flight reads worker.load(), no load report) with the expected wait breaking ties; least_requests samples no pair and scans every available worker (documented O(n) footnote); ties break uniformly at random in the shared scorer (least_load.rs:560-605). Retained claims hold: the no-report pair fallback to in-flight counts (least_load.rs:323-329) and default least_load tuning (factory.rs:22-26 builds LeastLoadPolicy::new()). The "newer than v1.12.0" note is backed by readable evidence: v1.12.0 is the newest tag in the source clone's packed-refs, the gateway crate on main is 1.12.0, and the feature commit is dated 2026-10-12, matching the repo's established release-framing convention. All edits serve the one concern (the same commit's --dp-aware validation change is correctly deferred to its own concern); ~45 changed lines. Placement is sound: both canonical pages corrected, no new page, new behavior in its own subsection/paragraph, release note standalone and stated once with the getting-started page linking to the canonical explanation, page summaries (overview card, complexity footnote, Load Signals bullet, advantages bullet) qualified, no code-change references, anchors resolve. Related pages retain no contradiction with this concern: reference/configuration.md:89 and concepts/architecture/overview.md:248 are one-line default-behavior summary rows matching those tables' idiom (their least_load rows likewise omit modifier flags), and the configuration reference's flag row belongs to tonight's separate deploy-config item, so completing this concern needs no page outside the allowlist. No feedback threads exist (retry after infrastructure failure), so none are reported addressed. Limitation: tag ancestry of commit 3f12f121 relative to v1.12.0 could not be walked without git tooling; the boundary rests on the tag list, the commit date, and repo precedent.

Workflow evidence

Human review threads and CODEOWNER approval remain under repository policy.