Skip to content

fix(dashboard): retry stalled events feed reconnects - #81931

Closed
helix4u wants to merge 1 commit into
NousResearch:mainfrom
helix4u:fix/dashboard-events-reconnect-stall
Closed

fix(dashboard): retry stalled events feed reconnects#81931
helix4u wants to merge 1 commit into
NousResearch:mainfrom
helix4u:fix/dashboard-events-reconnect-stall

Conversation

@helix4u

@helix4u helix4u commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Prevents the Dashboard events feed reconnect loop from stalling permanently after it has already displayed reconnecting in 1s....

The existing reconnect path only scheduled another attempt after a WebSocket close event. In gated Dashboard mode, every attempt first awaits buildWsUrl() to mint a fresh single-use ticket. If that request rejected or never settled, no replacement socket existed to emit close, so the retry loop stopped at its previous banner. A WebSocket handshake that emitted neither open nor close had the same failure mode.

This change gives the complete connection attempt, including ticket minting and the opening handshake, a 15-second deadline. Setup failures and deadline expiry feed back into the existing guarded exponential-backoff scheduler. A generation guard prevents late ticket responses or socket events from creating a superseded connection, and cleanup invalidates the active generation and clears both timers on unmount.

Related Issue

Discord support report: https://discord.com/channels/1053877538025386074/1535696700121948190

Follow-up to #79524.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • web/src/components/ChatSidebar.tsx
    • Catch URL/ticket construction failures and reschedule them through the existing reconnect policy.
    • Bound ticket minting plus the WebSocket opening handshake.
    • Invalidate timed-out attempts so late results cannot create competing sockets.
    • Clear the connection deadline on open, close, and unmount.
  • web/src/lib/events-reconnect.ts
    • Define the events-feed connection-attempt deadline alongside the existing reconnect policy constants.
  • web/src/components/ChatSidebar.test.tsx
    • Cover rejected URL construction, a stalled URL request with a late result, and a stalled WebSocket handshake.
    • Keep the existing error-plus-close single-scheduler regression explicit under the new handshake deadline.

How to Test

  1. Run cd web && npx vitest run src/components/ChatSidebar.test.tsx.
  2. Run cd web && npx tsc -p . --noEmit.
  3. Run cd web && npx vitest run.
  4. In a gated Dashboard session, interrupt the events-feed ticket request or opening handshake. Confirm the banner advances through backoff attempts instead of remaining at reconnecting in 1s... indefinitely.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass (not run: this is a web-only TypeScript change; the full Python suite is intentionally left to CI)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — N/A, no user-facing workflow changed
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A, no config keys changed
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A, existing reconnect architecture retained
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — browser timer and WebSocket APIs only
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A, no agent tools changed

Screenshots / Logs

Validation from the clean worktree:

  • Focused ChatSidebar.test.tsx: 16 tests passed.
  • Full web Vitest suite: 27 files, 194 tests passed.
  • Web TypeScript check: clean.
  • ESLint on the three changed files: 0 errors and one pre-existing react-refresh/only-export-components warning at ChatSidebar.tsx:104.
  • git diff --check: clean.

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/dashboard Web dashboard / control panel UI (dashboard/, landing) labels Aug 8, 2026
@helix4u
helix4u marked this pull request as ready for review August 8, 2026 18:55
@austinpickett

Copy link
Copy Markdown
Collaborator

Superseded by #81978.

Your diagnosis was right and the fix is carried there as the first commit, unchanged — the deadline around ticket minting plus the handshake, the generation guard against a late ticket opening a superseded socket, and all three tests. EVENTS_CONNECT_TIMEOUT_MS living next to the other policy constants in events-reconnect.ts was the right call.

The reason for the salvage rather than a straight merge: the bug class is "the await buildWsUrl() in front of new WebSocket() is unguarded, so a rejected or hanging ticket leaves nothing to emit close" — and that structure exists on ChatPage's PTY socket too, which is the main chat surface. PTY_CONNECTING_TIMEOUT_MS doesn't cover it, because that timer is armed after new WebSocket(url) returns; a ticket that never settles never arms it, and connectInFlightRef stays stuck true, which also blocks the page-resume recovery path. #81978 applies the same shape there and adds the mirrored tests.

You're credited via Co-authored-by on the salvage commit. Thanks for chasing this one down to the actual mechanism — the write-up on the ticket-minting gap is what made the sibling call path obvious.

austinpickett added a commit that referenced this pull request Aug 8, 2026
…supersedes #81931) (#81978)

* fix(dashboard): retry stalled events feed reconnects

* fix(dashboard): bound the PTY ticket request before the socket exists

ChatPage's connect awaits a single-use ticket from `api.buildWsUrl()`
before `new WebSocket()`. That request produces no socket, so a
rejection or a hang emits no `close` event and never arms
PTY_CONNECTING_TIMEOUT_MS (set after the socket is constructed). The
tab stranded on "connecting" with `connectInFlightRef` stuck true,
which also suppresses the page-resume reconnect path.

Give the ticket phase its own deadline and route both failure modes
into the existing backoff. A `ticketSuperseded` flag invalidates a late
ticket result so a timed-out attempt cannot open a socket behind the
replacement it scheduled, and cleanup clears the timer on unmount.

`scheduleReconnect` now takes `number | null` so an attempt that died
before any socket existed omits the "(code N)" banner suffix instead of
inventing one.

Same bug class as the events-feed fix in the preceding commit, on the
main chat surface.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

* test(dashboard): cover the PTY ticket connect deadline

Mirrors the events-feed cases in ChatSidebar.test.tsx: a rejected ticket
retries, a stalled ticket times out and its late resolution cannot open
a superseded socket, and a settled ticket disarms the deadline so
PTY_CONNECTING_TIMEOUT_MS remains the only guard on a wedged handshake
(NS-591 regression).

Both failure cases fail against ChatPage.tsx without the preceding fix.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

---------

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
ma1138569845 pushed a commit to ma1138569845/dechnicAuditor-agent that referenced this pull request Aug 10, 2026
…supersedes NousResearch#81931) (NousResearch#81978)

* fix(dashboard): retry stalled events feed reconnects

* fix(dashboard): bound the PTY ticket request before the socket exists

ChatPage's connect awaits a single-use ticket from `api.buildWsUrl()`
before `new WebSocket()`. That request produces no socket, so a
rejection or a hang emits no `close` event and never arms
PTY_CONNECTING_TIMEOUT_MS (set after the socket is constructed). The
tab stranded on "connecting" with `connectInFlightRef` stuck true,
which also suppresses the page-resume reconnect path.

Give the ticket phase its own deadline and route both failure modes
into the existing backoff. A `ticketSuperseded` flag invalidates a late
ticket result so a timed-out attempt cannot open a socket behind the
replacement it scheduled, and cleanup clears the timer on unmount.

`scheduleReconnect` now takes `number | null` so an attempt that died
before any socket existed omits the "(code N)" banner suffix instead of
inventing one.

Same bug class as the events-feed fix in the preceding commit, on the
main chat surface.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

* test(dashboard): cover the PTY ticket connect deadline

Mirrors the events-feed cases in ChatSidebar.test.tsx: a rejected ticket
retries, a stalled ticket times out and its late resolution cannot open
a superseded socket, and a settled ticket disarms the deadline so
PTY_CONNECTING_TIMEOUT_MS remains the only guard on a wedged handshake
(NS-591 regression).

Both failure cases fail against ChatPage.tsx without the preceding fix.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

---------

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…supersedes NousResearch#81931) (NousResearch#81978)

* fix(dashboard): retry stalled events feed reconnects

* fix(dashboard): bound the PTY ticket request before the socket exists

ChatPage's connect awaits a single-use ticket from `api.buildWsUrl()`
before `new WebSocket()`. That request produces no socket, so a
rejection or a hang emits no `close` event and never arms
PTY_CONNECTING_TIMEOUT_MS (set after the socket is constructed). The
tab stranded on "connecting" with `connectInFlightRef` stuck true,
which also suppresses the page-resume reconnect path.

Give the ticket phase its own deadline and route both failure modes
into the existing backoff. A `ticketSuperseded` flag invalidates a late
ticket result so a timed-out attempt cannot open a socket behind the
replacement it scheduled, and cleanup clears the timer on unmount.

`scheduleReconnect` now takes `number | null` so an attempt that died
before any socket existed omits the "(code N)" banner suffix instead of
inventing one.

Same bug class as the events-feed fix in the preceding commit, on the
main chat surface.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

* test(dashboard): cover the PTY ticket connect deadline

Mirrors the events-feed cases in ChatSidebar.test.tsx: a rejected ticket
retries, a stalled ticket times out and its late resolution cannot open
a superseded socket, and a settled ticket disarms the deadline so
PTY_CONNECTING_TIMEOUT_MS remains the only guard on a wedged handshake
(NS-591 regression).

Both failure cases fail against ChatPage.tsx without the preceding fix.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

---------

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
blut-agent pushed a commit to blut-agent/hermes-agent-fork that referenced this pull request Aug 11, 2026
…supersedes NousResearch#81931) (NousResearch#81978)

* fix(dashboard): retry stalled events feed reconnects

* fix(dashboard): bound the PTY ticket request before the socket exists

ChatPage's connect awaits a single-use ticket from `api.buildWsUrl()`
before `new WebSocket()`. That request produces no socket, so a
rejection or a hang emits no `close` event and never arms
PTY_CONNECTING_TIMEOUT_MS (set after the socket is constructed). The
tab stranded on "connecting" with `connectInFlightRef` stuck true,
which also suppresses the page-resume reconnect path.

Give the ticket phase its own deadline and route both failure modes
into the existing backoff. A `ticketSuperseded` flag invalidates a late
ticket result so a timed-out attempt cannot open a socket behind the
replacement it scheduled, and cleanup clears the timer on unmount.

`scheduleReconnect` now takes `number | null` so an attempt that died
before any socket existed omits the "(code N)" banner suffix instead of
inventing one.

Same bug class as the events-feed fix in the preceding commit, on the
main chat surface.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

* test(dashboard): cover the PTY ticket connect deadline

Mirrors the events-feed cases in ChatSidebar.test.tsx: a rejected ticket
retries, a stalled ticket times out and its late resolution cannot open
a superseded socket, and a settled ticket disarms the deadline so
PTY_CONNECTING_TIMEOUT_MS remains the only guard on a wedged handshake
(NS-591 regression).

Both failure cases fail against ChatPage.tsx without the preceding fix.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

---------

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
…supersedes NousResearch#81931) (NousResearch#81978)

* fix(dashboard): retry stalled events feed reconnects

* fix(dashboard): bound the PTY ticket request before the socket exists

ChatPage's connect awaits a single-use ticket from `api.buildWsUrl()`
before `new WebSocket()`. That request produces no socket, so a
rejection or a hang emits no `close` event and never arms
PTY_CONNECTING_TIMEOUT_MS (set after the socket is constructed). The
tab stranded on "connecting" with `connectInFlightRef` stuck true,
which also suppresses the page-resume reconnect path.

Give the ticket phase its own deadline and route both failure modes
into the existing backoff. A `ticketSuperseded` flag invalidates a late
ticket result so a timed-out attempt cannot open a socket behind the
replacement it scheduled, and cleanup clears the timer on unmount.

`scheduleReconnect` now takes `number | null` so an attempt that died
before any socket existed omits the "(code N)" banner suffix instead of
inventing one.

Same bug class as the events-feed fix in the preceding commit, on the
main chat surface.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

* test(dashboard): cover the PTY ticket connect deadline

Mirrors the events-feed cases in ChatSidebar.test.tsx: a rejected ticket
retries, a stalled ticket times out and its late resolution cannot open
a superseded socket, and a settled ticket disarms the deadline so
PTY_CONNECTING_TIMEOUT_MS remains the only guard on a wedged handshake
(NS-591 regression).

Both failure cases fail against ChatPage.tsx without the preceding fix.

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>

---------

Co-authored-by: Gille <4317663+helix4u@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants