Skip to content

fix(desktop): full-jitter backoff on gateway WS reconnect (salvage #76282) - #77654

Merged
kshitijk4poor merged 2 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/76282-full-jitter-backoff
Aug 3, 2026
Merged

fix(desktop): full-jitter backoff on gateway WS reconnect (salvage #76282)#77654
kshitijk4poor merged 2 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/76282-full-jitter-backoff

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Context

Every desktop client that loses its gateway WebSocket redials on the same deterministic ladder (1s·2^n capped at 15s in two places, 30s in a third). A gateway restart makes every open desktop window redial in lockstep — the thundering-herd shape full jitter exists to break. WHO: anyone running multiple desktop windows/machines against one gateway, most visibly during gateway restarts/updates.

This centralizes the scattered backoff policy into one resolver (lib/reconnect-backoff.ts, AWS full-jitter: random[0, min(cap, base·2^n))) and applies it to all three reconnect sites — matching the desktop guide's "one resolver owns each policy" rule.

Provenance and the escalation fix

Salvage of #76282 (author trevornk; attribution mapping merged as #77641). Stage-1 review found a real blocker in the original: RECONNECT_ESCALATE_AFTER = 6 counted ATTEMPTS, calibrated to the old deterministic ladder (~45s of failures before surfacing "Lost connection"). Full jitter reaches 6 attempts in ~9s expected — so a brief network blip would flash a boot error. An existing test caught it deterministically. The follow-up commit switches escalation to elapsed time since first failure (constant calibrated to the old ladder's wall-clock behavior), making that existing test pass UNCHANGED — the test is the contract.

Measured impact

Behavioral: jitter bounds verified [0, min(cap, base·2^n)), no overflow; escalation timing preserved at the old ladder's wall-clock calibration. Herd-dispersion is probabilistic by construction (full jitter is the AWS-recommended variant for exactly this).

Verification

  • Full use-gateway-boot.test.tsx suite + store/gateway tests green on the final stack, including the post-boot-blip contract test that failed on the unfixed pick.
  • Mutation checks: de-jittering the backoff fails the jitter test; reverting the escalation fix makes the post-boot-blip test fail again; restore → green.

Closes #76282.

Hermes Coder and others added 2 commits August 3, 2026 17:35
All three desktop reconnect loops (primary gateway boot, secondary
multi-profile gateway pool, plugin event socket) used bare exponential
backoff with no jitter. After a gateway restart every disconnected
client redials on the exact same schedule, so the reconnect attempts
land in lockstep instead of spreading out -- a burst that can starve
the gateway's file descriptors while it's still coming back up.

Add reconnect-backoff.ts implementing AWS-style full-jitter backoff
(random delay in [0, min(cap, base * 2^attempt))) and wire it into all
three call sites in place of their local Math.min/2**attempt math.
Manual reconnect paths already reset the attempt counter and bypass
the timer entirely -- unchanged.
… count

With the full-jitter backoff (300ms base) six attempts can elapse in ~9s,
so the old RECONNECT_ESCALATE_AFTER=6 attempt threshold raised the
recoverable boot error during a brief post-boot blip — breaking the
'a remote that drops post-boot keeps looping with NO boot.error' contract.
Escalate after RECONNECT_ESCALATE_AFTER_MS (45s, matching the old
deterministic 1->15s ladder's calibration) elapsed since the first failed
reconnect of the episode. Reset on clean open, manual/wake reconnect, and
soft switch, preserving the reset-on-success path.
@kshitijk4poor
kshitijk4poor enabled auto-merge (rebase) August 3, 2026 12:40
@kshitijk4poor
kshitijk4poor merged commit 5ffbea8 into NousResearch:main Aug 3, 2026
34 checks passed
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/desktop Electron desktop app (apps/desktop/*) labels Aug 3, 2026
@kshitijk4poor
kshitijk4poor deleted the salvage/76282-full-jitter-backoff branch August 5, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants