fix(desktop): prevent false runtime-not-ready under gateway load

[UnathiCodex]

What does this PR do?

Hermes Desktop polls REST status and gateway runtime readiness every 15 seconds. Under heavy concurrent agent load, session.active_list could occupy the shared WebSocket reader for tens of seconds. Readiness requests queued behind it could then time out, and the Desktop hook published the transport fallback (ready: false) as if it were an authoritative credential failure. The interval could also start overlapping refreshes, allowing a stale failure to race a newer healthy result.

This PR fixes both sides of that failure mode:

• Desktop status refreshes are single-flight and schedule the next poll only after the current refresh settles.
• REST and readiness results settle independently.
• Transient readiness fallbacks preserve the last authoritative result; a real authoritative ok: false is still surfaced immediately.
• session.active_list runs through the gateway worker pool so a slow live-session rehydrate cannot block readiness, prompts, or interrupts on the same socket.

Related Issue

No linked issue. Reproduced against a live remote Desktop/backend connection under concurrent agent load.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✅ Tests (adding or improving test coverage)

Changes Made

• Updated apps/desktop/src/app/shell/hooks/use-status-snapshot.ts to use non-overlapping, independently settled polling.
• Added apps/desktop/src/app/shell/hooks/use-status-snapshot.test.ts covering transient failures, real credential failures, initial unknown state, and overlapping polls.
• Added session.active_list to tui_gateway.server._LONG_HANDLERS.
• Extended the gateway GIL-starvation regression matrix.

How to Test

1. Connect Desktop to a remote Hermes dashboard and start several concurrent agent turns.
2. Keep the connection under enough load that live-session polling is delayed.
3. Verify Desktop retains the last authoritative readiness during the transient delay, still reports genuine credential failures, and continues polling without overlapping refreshes.

Automated checks run:

• npm test -- src/app/shell/hooks/use-status-snapshot.test.ts src/lib/runtime-readiness.test.ts — 10 passed
• npm run typecheck — passed
• targeted ESLint — passed
• scripts/run_tests.sh tests/tui_gateway/test_inline_rpc_gil_starvation.py — 9 passed on Ubuntu

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've run the complete Python test suite
• I've added tests for my changes
• I've tested on Windows 11 Desktop and an Ubuntu remote backend

Documentation & Housekeeping

• Documentation changes are N/A
• Config example changes are N/A
• Contributor/agent guide changes are N/A
• I've considered cross-platform impact
• Tool description/schema changes are N/A

Screenshots / Logs

During the live failure, setup.runtime_check still returned ok: true for the configured provider while session.active_list took 23.2 seconds on the same connection. After applying both fixes, live-session polling completed in 209 ms and authoritative readiness returned ok: true in 337 ms.

[alt-glitch]

This was generated by AI during triage.

Related to #65153 and #65388. The live branches now have a documented ownership split: this PR owns Desktop readiness-state polling plus session.active_list worker-pool routing; #65153 owns readiness-probe singleflight; and #65388 retains the distinct model.options routing and stall-watchdog work. The former overlapping polling slice was removed from #65388.

[UnathiCodex]

Thanks for flagging this. I compared the actual diffs in #65388 and the runtime-probe work described by #65151 / #65153.

Proposed ownership split:

• fix(desktop): prevent false runtime-not-ready under gateway load #66174 owns the Desktop readiness-state semantics plus the directly observed session.active_list reader-thread blockage.
• [codex] fix(gateway): singleflight runtime readiness probes #65153 owns server-side singleflight/deduplication of concurrent setup.runtime_check probes. That is complementary rather than duplicate.
• fix(desktop): harden stalled gateway recovery #65388 can retain its distinct model.options routing and in-stall traceback watchdog work, while dropping the overlapping useStatusSnapshot slice (or otherwise taking the stronger polling implementation from this PR).

The Desktop implementations are not quite equivalent: #66174 independently settles REST status and readiness with Promise.allSettled, and schedules the next poll only after the current refresh settles. This means a transient failure in one leg cannot suppress a valid update from the other, and old/slow polls cannot overlap. #65388 currently keeps the combined Promise.all plus interval/in-flight guard.

The backend delta here is also unique to the live reproduction: while the warning was visible, setup.runtime_check remained authoritative ok: true (~301 ms), but session.active_list occupied the shared WS reader for 23.2 s. Pool-routing that handler reduced the post-fix live check to ~209 ms. #65388 and #65153 do not currently route session.active_list.

All required CI on #66174 is green. I am happy to coordinate with @yingliang-zhang and rebase/drop any truly duplicate slice once the ownership split is agreed; the goal is one canonical implementation, not competing readiness patches.

[tonydwb]

Code Review Summary

Verdict: Approved

Desktop fix preventing false runtime-not-ready detection under gateway load. The diff looks like a state management fix in the React renderer. No security concerns.

---

Reviewed by Hermes Agent

[yingliang-zhang]

Thanks @c3llus — I agree with the ownership split.

I've narrowed #65388 to its non-overlapping pieces only: model.options pool routing with a coherent request-time live-runtime snapshot, plus the in-stall faulthandler traceback watchdog. The overlapping useStatusSnapshot implementation and tests have been removed from #65388. #65153 can continue to own readiness-probe singleflight, while this PR owns the Desktop polling loop and session.active_list routing.

One behavior from the former #65388 polling tests is worth preserving here: when gatewayState leaves open, readiness should clear immediately without waiting for getStatus() to settle. In the current implementation, the null update occurs only after Promise.allSettled(...); if the REST status request hangs during a gateway disconnect, the UI can retain a stale authoritative ready state. A focused regression can hold getStatus() pending, rerender from open to connecting, and assert inferenceStatus becomes null synchronously.

Otherwise, the Promise.allSettled + serial setTimeout loop here is the stronger owner for the polling behavior because one failed leg cannot suppress a successful result from the other.

#65388 narrowed head: 9711506d53974c152501fcbfc0002606598e57e7.

[UnathiCodex]

Thanks @yingliang-zhang — implemented the disconnect-state regression you described in 8f15a5408. The hook now clears inferenceStatus as soon as gatewayState leaves open, before awaiting the REST status leg, so a hung getStatus() cannot preserve stale readiness. The focused test first establishes authoritative ready, holds the reconnect getStatus() pending, rerenders open → connecting, and asserts the readiness state is immediately null.

I also rebased #66174 onto current main. Focused readiness tests pass (11/11), along with Desktop typecheck, targeted ESLint, Prettier, and git diff --check.

[teknium1]

Thanks for the focused readiness and reader-path fix. The current-main premise is confirmed: apps/desktop/src/app/shell/hooks/use-status-snapshot.ts:20-44 couples the legs and permits overlapping polls, while tui_gateway/server.py:1451-1453 still dispatches session.active_list inline because it is absent from _LONG_HANDLERS.

Problems

• scripts/release.py:45 adds a new email to the legacy map. Current main marks that map frozen at scripts/release.py:46-50; contributors/README.md:1-24 requires new mappings to be one-file-per-email entries under contributors/emails/. This portion will conflict on salvage and is no longer the supported attribution mechanism.

Suggested changes

• Replace that edit with contributors/emails/theunathi@gmail.com containing UnathiCodex as its first non-comment line.

Automated hermes-sweeper review.

[teknium1]

Current main freezes the legacy map and requires new mappings under contributors/emails/ (contributors/README.md). Please replace this with contributors/emails/theunathi@gmail.com containing UnathiCodex; otherwise this will conflict on salvage and bypass the conflict-free mapping mechanism.

[austinpickett]

Review

Verified locally on Windows 11 (worktree at PR head 48f01e2ef, plus a trial merge with current main).

What I checked

Premise verification — traced the failure mode against the actual code:

• On the merge-base, use-status-snapshot.ts used setInterval + a single Promise.all with a .catch that converted any readiness rejection into {ready: false, source: 'fallback'} and published it unconditionally. So a queued/timed-out RPC really did surface as "inference not ready", and setInterval really did allow overlapping refreshes where a stale failure could land after a newer healthy result. Premise holds.
• session.active_list was indeed absent from _LONG_HANDLERS in tui_gateway/server.py, so it ran inline on the WS reader thread — consistent with the reported 23s stall blocking readiness RPCs queued behind it on the same socket. Adding it follows the exact precedent of the setup.runtime_check/setup.status entries (#50005 family) directly above it.

Design of the fix:

• Promise.allSettled + independent settling is right: a hung getStatus() no longer discards a good readiness result and vice versa.
• The source !== 'fallback' gate is the correct authoritative/transient discriminator — runtime-readiness.ts only returns source: 'fallback' when neither RPC produced a boolean, so suppressing it preserves the last authoritative state without masking a real ok: false (which arrives as source: 'runtime_check' and is still published immediately, covered by a test).
• Chained setTimeout scheduled in finally gives single-flight polling with no overlap and no dropped schedule on error paths; cleanup handles both the interval→timeout switch and the cancelled flag.
• Clearing readiness when gatewayState !== 'open' closes the complementary staleness hole (stale "ready" surviving a disconnect) — good catch to handle both directions.

Tests: the 5 new hook tests cover transient failure retention, initial-failure → null (no false negative on first paint), authoritative failure propagation, disconnect clearing, and non-overlap of slow refreshes. These are behavior contracts, not change-detectors. The GIL-starvation matrix addition means the _LONG_HANDLERS entry is regression-guarded, not just asserted once.

Local results (Windows 11)

• npx vitest run src/app/shell/hooks/use-status-snapshot.test.ts src/lib/runtime-readiness.test.ts — 11 passed
• npm run typecheck (apps/desktop, both tsconfigs) — clean
• pytest tests/tui_gateway/test_inline_rpc_gil_starvation.py — 9 passed
• Trial merge with current main — clean auto-merge, no semantic conflict with the recent _reset_session_agent changes in tui_gateway/server.py

CI is fully green and the branch merges cleanly. Nice, tightly-scoped fix that addresses both sides of the failure mode (frontend polling semantics + backend reader-thread starvation) with a clear line-level account of each. LGTM — approving.