feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0) - #52243
Merged
Conversation
Net-new WebSocketRelayTransport.go_dormant() + RelayAdapter.go_dormant() — the third transport mode the scale-to-zero behaviour layer needs, distinct from both disconnect() and an unexpected close (decisions.md D12/F14): - disconnect() sets _closing=True and CANCELS the reconnect supervisor (terminal "shutting down for good") -> a suspended machine never re-dials on wake, stranding its buffered backlog. - an unexpected close re-dials IMMEDIATELY -> the socket never stays down, so the platform proxy never suspends the machine. go_dormant(): going_idle->ack (reuse go_idle), then close the socket WITHOUT setting _closing, so the reader's fall-through still arms the reconnect supervisor (wake path stays live) but on the longer _dormant_redial_s cadence so it doesn't fight the platform suspend window. A successful re-dial clears _dormant. Honors the §3.4 wake->reconnect->drain contract. Tests: 6 new in test_relay_going_idle.py incl. the F14 regression guard (routing dormancy through disconnect() fails exactly the 4 wake-path tests). Full relay suite 140 passed.
The gateway-side BEHAVIOUR layer that consumes the relay scale-to-zero primitives (gateway-gateway Phase 5): the gateway decides it is idle and drives the relay transport dormant so the platform (Fly autostop:"suspend") can suspend the now-traffic-idle machine, which wakes on the connector's wakeUrl poke (decisions.md Q3=C', D1-D13). - gateway/scale_to_zero.py: pure helpers — scale_to_zero_enabled (the NAS Labs HERMES_SCALE_TO_ZERO stamp, D11/Q8=A), parse_idle_timeout_seconds (config.yaml gateway.scale_to_zero.idle_timeout_minutes, D2), messaging_is_relay_only_or_absent (F6/D1), should_arm (D1/D11/§3.4(1)), is_idle (D2/D3/F7). - gateway/run.py: _last_inbound_at clock stamped on user inbound in _handle_message (F13); the arm-gate + idle predicate + the _scale_to_zero_watcher dormant sequence (mark draining -> adapter go_dormant() -> cooldown), started only when armed. Deliberately NOT the stop path and NOT mark_resume_pending (F12/D13). - tools/process_registry.py: has_any_active() for the bg-work guard (D3/F7). - hermes_cli/config.py: gateway.scale_to_zero.idle_timeout_minutes default 5. Tests: 38 pure-logic + 6 watcher (incl. bg-work regression guard proven RED). Full relay + scale-to-zero suites: 184 passed. The 20 unrelated failures in the broader run are PRE-EXISTING on origin/main (custom-provider/tools tests), confirmed via a pristine baseline worktree.
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-import |
2 |
invalid-argument-type |
1 |
First entries
tests/gateway/relay/test_relay_going_idle.py:434: [invalid-argument-type] invalid-argument-type: Argument to `RelayAdapter.__init__` is incorrect: Expected `RelayTransport | None`, found `_StubTransport`
tests/gateway/test_scale_to_zero.py:11: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/gateway/test_scale_to_zero_watcher.py:16: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
✅ Fixed issues: none
Unchanged: 5937 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
alt-glitch
added
type/feature
1 task
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Salvage of #52207 onto current
main. Phase 0 of scale-to-zero: the gateway-side behaviour layer that lets a relay-only hosted instance suspend while idle and wake on demand without dropping inbound, consuming the Phase 5 relay primitives already onmain(#51572, #51595).The gateway decides it is idle, drains the relay via a new
go_dormant()transport mode, and goes dormant so the platform (Flyautostop:"suspend") can suspend the now-traffic-idle machine; it wakes on the connector's wakeUrl poke and the reconnect supervisor drains the buffered backlog. The gateway never calls a vendor API — platform suspend/wake is the NAS/Fly layer's job (a later phase).What's in this PR (Phase 0, gateway-side only)
go_dormant()— a third relay close mode. Distinct fromdisconnect()(terminal — cancels the reconnect supervisor) and an unexpected close (re-dials immediately): sendsgoing_idle→ack then closes the socket while preserving the reconnect supervisor so the wake path stays armed. The supervisor then polls on a dormant cadence rather than fighting the suspend window.gateway/scale_to_zero.py— pure, unit-testable idle/arm logic. Idle = no in-flight agent turn AND no inbound for N minutes (config, default 5) AND no live background work (backgrounded subagents / kanban /terminal(background=true)).HERMES_SCALE_TO_ZEROLabs stamp), messaging is relay-only or absent, and a wakeUrl is registered. A non-opted instance behaves exactly as today._handle_messagechokepoint (user-originated inbound only).process_registry.has_any_active()— cross-session live-process check for the background-work conjunct.Changes vs the original PR
peer: truechurn inpackage-lock.json(npm-version artifact); diff is now exactly the 9 substantive files.Validation
maintest_scale_to_zero,test_scale_to_zero_watcher,test_relay_going_idle)py_compileOriginal PR #52207 by @benbarclay. Authorship preserved via rebase-merge.
Infographic