docs: A worker whose pushed load records go silent is routed around after --worker-stale-secs (default 15 s)

[github-actions]

What this PR does

When a gRPC or ZMQ worker that has been pushing load records goes completely silent (dead link, one-way cut), how long until the gateway stops routing to it, which workers does the --worker-stale-secs rule judge, what clears the exclusion, and how is it tuned or disabled?

Why we need it

Source change: smg-project/smg@2a482c3

Commit 217 (#2988, 2a482c36, 2026-10-09, 'a liveness veto for a worker whose pushed load records stopped'); its supplied diff was read and every claim verified on the current default-branch snapshot. model_gateway/src/worker/liveness.rs — records_stopped (:543, verified current) vetoes a worker as unreachable when it has pushed load records on its KV-event stream and nothing has been heard from it on any channel for the stale threshold (reason logged 'load records stopped', applied by the 250 ms sweep at :340 and :363); the module doc pins the contract: a reachable servicer pushes a record at least every few seconds (a record rides every event batch and a heartbeat fills every silence), so a longer total silence is a dead or one-way link the transport has not reported yet (a one-way cut leaves the connection open until the keepalive verdict tens of seconds later, and a failed poll deadline is deliberately not a transport failure); workers that never pushed a record (HTTP workers, older servicers, a stream still connecting) are not judged by their silence — a poll every ten seconds and a probe every minute is all that is expected of them; any contact clears the veto (a load record, a poll answer, a probe, a token — a worker still streaming tokens is never stale); the veto never touches the worker's health status. Threshold: --worker-stale-secs, default 15 (DEFAULT_STALE liveness.rs:103 — three idle five-second heartbeats and past the ten-second load poll, so a stream that stopped while the poll still answers is never judged by its silence alone), 0 disables; keep it above --load-monitor-interval (model_gateway/src/main.rs flag help, help_heading Load Monitoring); the Python launcher matches (bindings/python/src/smg/router_args.py worker_stale_secs = 15 with the same help). Tests pin it: records_stopped_needs_a_pushing_worker_silent_for_the_threshold and a_pushing_worker_whose_records_stop_is_vetoed_and_the_next_contact_restores_it (liveness.rs:765+). Missing documentation: a repo-wide grep for worker-stale-secs and 'load records' in src/lib/content returns nothing; concepts/reliability/overload-protection.md:111-115 ('When a Worker Does Not Report — Overload protection fails open ... the worker stays routable') states the fail-open side with no mention that a pushing worker whose contact stops entirely is excluded as unreachable after 15 s, and its :102 ZMQ row already explains load records riding the engine's output batches — this page's Load Monitoring material is the docs' home for load-record-driven behavior, so the rule lands here as a short subsection beside the fail-open section, keeping the two rules clearly distinct (no report = no overload opinion; no contact at all from a pusher = unreachable). Placement alternatives examined: concepts/reliability/health-checks.md (reachability) makes no claim this rule contradicts, is driven by probe machinery the veto is explicitly independent of, and is consumed by my breaker item tonight. Scope limits: the sibling stall and wedge vetoes (--worker-stall-secs/--worker-wedge-secs) are also undocumented but predate this commit and need their own attribution (recorded in remaining_work) — this edit documents only the stale-records rule, naming the others at most in passing; the configuration.md flag row is the deploy-config scan's. Version framing: #2988 (2026-10-09) postdates the v1.12.0 bump (commit 299, #2888, 2026-10-08); present as current-main behavior newer than v1.12.0 — before it, a one-way cut was only caught by transport failures or keepalive verdicts. One independent concern: one commit added one routing-exclusion rule and its one flag, documented beside the page section whose fail-open claim it qualifies. Dedup: no open or closed PR mentions #2988, worker-stale-secs, or stale load records (closed-unmerged #78 covered the load_monitor_interval_secs spec field, a different question); open PRs #144-150 do not hold overload-protection.md; my other items do not touch it. Edit well under 1,000 lines.

Scope: worker-liveness / stale-load-records-veto. Other concerns are deferred.

How to test

• Passed the documentation path and size guard (under 1000 added plus deleted lines; no file-count limit).
• Passed an independent accuracy and single-concern review.
• Passed git diff --check and pnpm check and the production pnpm build.

Checklist

• Tests added/updated (if applicable)
• Docs updated (if applicable)
• Commit includes a DCO sign-off

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: eccd75a1-4dd0-4ad3-a0c2-d5880557d777

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Documentation maintenance: ready. Unsuccessful content rounds: 0/3. Operational attempts: 0/3.

Head: 277b5096629652b1e1f2288f4b21c43dbfebe63b; reviewed main: 8df4f5b4b0429633d2a22f92c67a2c93640477d5.
Reviewed SMG source: a29c64d4323dbf21950e2822cb099af04f6fd901.

All edits serve the stale-load-records-veto concern on the single allowlisted canonical page. Every claim verified against source a29c64d4: --worker-stale-secs default 15 / 0 disables (main.rs:681, liveness.rs:111, records_stopped liveness.rs:598, router_args.py:309); only gRPC KV-event-stream workers ever get note_load_record (sole caller apply_pushed_load monitor.rs:499 from subscription.rs:437; SubscribeKvEvents unimplemented over ZMQ, backend_client.rs:243), so the HTTP/ZMQ exemption is correct; contact sources (poll answers monitor.rs:1125, passed probes on_probe_passed, KV batches/records, tokens via note_token_progress updating last_contact_ms) all clear the Unreachable veto at once; the 250 ms sweep (monitor.rs:1315) runs unconditionally for all workers and the text correctly avoids an exact-instant promise. The prior review's counterexample is repaired exactly: the steering bullet now matches fallback_if_all_vetoed (overload.rs:115) — least-loaded healthy, breaker-closed candidate that shedding does not rule out, with shed_if_all_overloaded running first under --worker-overload-shed. The invalid release comparison was replaced by a short standalone Unreleased note with no version boundary; the rewritten section drops log strings and stream plumbing while keeping all required elements (~270 words vs the ~200 target, acceptable). Fail-open generalization qualified in place; no code/PR references in prose. Related pages (health-checks.md, configuration.md, zmq-workers.md, kv-events-cache-aware.md, reliability index) retain no contradictory claims; the configuration.md flag row is assigned to the deploy-config scan and the table claims no completeness. Limitation: the page's pre-existing overload-protection defaults/shedding prose is stale against current main (protection now on by default, shed opt-in), but that is untouched text constituting a separate concern; the new section stays accurate independently. No review threads exist in the feedback payload, so addressed_threads is empty; the state-level repair guidance is fully reflected in the final prose.

Workflow evidence

Human review threads and CODEOWNER approval remain under repository policy.

[XinyueZhang369]

This source change is not included in the latest stable release, v1.12.0. Public documentation will now target published stable releases only, without unreleased/current-main sections. Closing at the maintainer’s request. The docs-awaiting-release label marks this as deferred rather than declined, so the nightly workflow can rediscover the gap against a stable release and propose a focused, release-correct update.