[Router] Treat an upstream 503/429 as backpressure, not a breaker fault (2/3)

[Kangyan-Zhou]

Motivation

2 of 3 in the router request-attribution stack. #39463 (1/3) has
merged, so this now targets main directly. #39465 (3/3) makes
worker_requests_total{outcome} agree with the judgement this PR teaches the
circuit breaker.

A saturated engine answers with its own queue-full 503s. The router counts each
one as a worker fault, and after the failure threshold the circuit breaker
opens. With a single worker behind the router that sheds every request for
the whole cool-down — and it keeps shedding after the engine has drained and
gone idle, because nothing is being sent to notice that it recovered.

The signal is backwards at exactly the wrong moment: the response that means
"I am alive and busy" is treated as evidence the worker is dead.

Modifications

A new breaker_outcome classifies the upstream status into three
dispositions, applied identically on the JSON and the streaming dispatch arm:

upstream status	outcome	breaker call
any non-5xx — 2xx, and a 4xx the worker answered cleanly	Success	record_success (streaming defers to the pump's completion hook)
503, 429	Neutral	record_backpressure
other 5xx	Failure	record_failure

Transport errors, timeouts, mid-body drops, a malformed discovery URL, and a
stream whose body dies after a 2xx head are all recorded as failures at their
call sites, unchanged.

CircuitBreaker::record_backpressure is the new third verb, and its
behaviour is per-state rather than a blanket no-op:

• Closed: do nothing, and specifically do not reset the failure streak
  the way record_success does. A worker interleaving genuine 5xx faults with
  503s still trips.
• HalfOpen: close. Any response observed here proves the worker is
  answering, which is what the probe exists to find out. Leaving HalfOpen
  unresolved would wedge the breaker permanently — the probe slot is released
  only by a success or a failure, and backpressure is neither — shutting a
  recovered-but-busy worker out forever, a worse false-shed than the one this
  PR removes.
• Open: no-op. This state is reachable, not dead code: allow gates
  admission, not completion, so a request admitted while Closed can return
  after concurrent failures have opened the breaker. A late 503 must not reset
  a breaker that has already tripped, exactly as record_failure ignores
  failures while Open.

The tradeoff is stated at breaker_outcome, not hidden: a worker stuck
returning 503 forever is no longer detected here. HTTP status alone cannot tell
"busy" from "broken-and-saying-503", and counting it as a fault produced the
worse, fleet-wide false-shed. Detecting a wedged-at-503 worker needs a
different signal than this one.

Tests

Three fail without the change, at both levels:

• Breaker-level: probe resolution from HalfOpen, failure-streak preservation
  while Closed, and backpressure alone never opening a Closed breaker.
• End-to-end through forward_json_to / forward_streaming_to against a live
  mock worker: an engine answering 503 far past any plausible threshold leaves
  the breaker Closed and admitting; a 500 still opens it; a half-open probe
  answered with 503 recovers the breaker.

The threshold tests loop on would_allow() rather than a hard-coded count, so
a change to the default CircuitBreakerConfig cannot silently make them
vacuous.

cargo test green on this branch — 725 lib, 7 bin, 65 component, 121 proxy.
cargo clippy --all-targets -- -D warnings clean; pre-commit clean.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation — N/A, no user-facing configuration surface; the dispositions and the tradeoff are documented at breaker_outcome and record_backpressure.
• Provide accuracy and speed benchmark results — N/A, one status classification per dispatch.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

https://claude.ai/code/session_012xa7Ey1ujKRSrfw4CMrAF5

---

CI States

Latest PR Test (Base): ✅ Run #35324927247
Latest PR Test (Extra): ❌ Run #35324927049
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[ShangmingCai]

/tag-and-rerun-ci