Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion src/lib/content/concepts/reliability/overload-protection.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,23 @@ The gateway's load monitor collects one report per worker per poll. The source d

### When a Worker Does Not Report

Overload protection **fails open**. A poll that produces no report (an unsupported backend, a timeout, an error, or an empty report) clears the worker's veto: no fresh signal means no opinion, and the worker stays routable. The veto is also cleared when a worker leaves `Ready`, is removed, or is replaced (`PUT` or `PATCH` on `/workers/{worker_id}`), and for every worker when the monitor has to rebuild its state (logged as a warning).
Overload protection **fails open**. A poll that produces no report (an unsupported backend, a timeout, an error, or an empty report) clears the worker's veto: no fresh signal means no opinion, and overload protection on its own leaves the worker routable. Whether the worker is reachable at all is judged separately; see [When a Worker Goes Completely Silent](#when-a-worker-goes-completely-silent). The veto is also cleared when a worker leaves `Ready`, is removed, or is replaced (`PUT` or `PATCH` on `/workers/{worker_id}`), and for every worker when the monitor has to rebuild its state (logged as a warning).

A field the backend leaves out reads as `0`, so that signal never trips. `--worker-overload-token-usage` only works for backends that report KV usage.

### When a Worker Goes Completely Silent

Failing open covers a missing report, not a missing worker. A gRPC worker that has pushed load records on its KV-event stream is expected to stay in contact: when nothing is heard from it on any channel — no load record, no answered load poll, no passed health probe, no token or completion — for `--worker-stale-secs` seconds (default `15`), the gateway excludes it from routing as unreachable. A periodic check applies the exclusion once the threshold has passed, not at the exact instant it elapses.

- **Only workers that have pushed a load record are judged.** HTTP and ZMQ workers, gRPC workers whose servicer does not push load records, and a stream that is still connecting are never excluded for silence; answering the load poll and the health probes is all that is expected of them.
- **Any contact clears the exclusion.** The next load record, answered poll, passed health probe, token, or completion re-admits the worker at once. A worker that is still streaming tokens is never considered silent.
- **The exclusion steers rather than refuses.** It only applies to selection: requests go to the other candidates while it lasts, and requests already running on the worker are unaffected. Even when this rule has excluded every candidate, the request is still routed to the least-loaded of them that is healthy, whose circuit breaker is not open, and that overload [shedding](#shedding-requests) does not rule out. Silence alone never rejects a request; shedding is independent of this rule, and a request whose every candidate is also overloaded can still be shed rather than steered.
- **Independent of overload protection.** The rule runs whether or not overload protection is enabled, and it never changes the worker's health status — the health checker keeps its own state machine.

Keep `--worker-stale-secs` above `--load-monitor-interval`: the default `15` is past the default ten-second poll, so a worker whose records stop while it still answers polls is never excluded for silence alone. `0` disables the rule. The Python launchers accept the flag too.

Unreleased: this rule is not yet part of a tagged SMG release.

---

## Configuration
Expand Down