fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf - #13231
Merged
Conversation
Long TUI sessions were crashing Node via V8 fatal-OOM once transcripts +
reasoning blobs crossed the default 1.5–4GB heap cap. This adds defense
in depth: a bigger heap, leak-proofing the RPC hot path, bounded
diagnostic buffers, automatic heap dumps at high-water marks, and
graceful signal / uncaught handlers.
## Changes
### Heap budget
- hermes_cli/main.py: `_launch_tui` now injects `NODE_OPTIONS=
--max-old-space-size=8192 --expose-gc` (appended — does not clobber
user-supplied NODE_OPTIONS). Covers both `node dist/entry.js` and
`tsx src/entry.tsx` launch paths.
- ui-tui/src/entry.tsx: shebang rewritten to
`#!/usr/bin/env -S node --max-old-space-size=8192 --expose-gc` as a
fallback when the binary is invoked directly.
### GatewayClient (ui-tui/src/gatewayClient.ts)
- `setMaxListeners(0)` — silences spurious warnings from React hook
subscribers.
- `logs` and `bufferedEvents` replaced with fixed-capacity
CircularBuffer — O(1) push, no splice(0, …) copies under load.
- RPC timeout refactor: `setTimeout(this.onTimeout.bind(this), …, id)`
replaces the inline arrow closure that captured `method`/`params`/
`resolve`/`reject` for the full 120 s request timeout. Each Pending
record now stores its own timeout handle, `.unref()`'d so stuck
timers never keep the event loop alive, and `rejectPending()` clears
them (previously leaked the timer itself).
### Memory diagnostics (new)
- ui-tui/src/lib/memory.ts: `performHeapDump()` +
`captureMemoryDiagnostics()`. Writes heap snapshot + JSON diag
sidecar to `~/.hermes/heapdumps/` (override via
`HERMES_HEAPDUMP_DIR`). Diagnostics are written first so we still get
useful data if the snapshot crashes on very large heaps.
Captures: detached V8 contexts (closure-leak signal), active
handles/requests (`process._getActiveHandles/_getActiveRequests`),
Linux `/proc/self/fd` count + `/proc/self/smaps_rollup`, heap growth
rate (MB/hr), and auto-classifies likely leak sources.
- ui-tui/src/lib/memoryMonitor.ts: 10 s interval polling heapUsed. At
1.5 GB writes an auto heap dump (trigger=`auto-high`); at 2.5 GB
writes a final dump and exits 137 before V8 fatal-OOMs so the user
can restart cleanly. Handle is `.unref()`'d so it never holds the
process open.
### Graceful exit (new)
- ui-tui/src/lib/gracefulExit.ts: SIGINT/SIGTERM/SIGHUP run registered
cleanups through a 4 s failsafe `setTimeout` that hard-exits if
cleanup hangs.
`uncaughtException` / `unhandledRejection` are logged to stderr
instead of crashing — a transient TUI render error should not kill
an in-flight agent turn.
### Slash commands (new)
- ui-tui/src/app/slash/commands/debug.ts:
- `/heapdump` — manual snapshot + diagnostics.
- `/mem` — live heap / rss / external / array-buffer / uptime panel.
- Registered in `ui-tui/src/app/slash/registry.ts`.
### Utility (new)
- ui-tui/src/lib/circularBuffer.ts: small fixed-capacity ring buffer
with `push` / `tail(n)` / `drain()` / `clear()`. Replaces the ad-hoc
`array.splice(0, len - MAX)` pattern.
## Validation
- tsc `--noEmit` clean
- `vitest run`: 15 files, 102 tests passing
- eslint clean on all touched/new files
- build produces executable `dist/entry.js` with preserved shebang
- smoke-tested: `HERMES_HEAPDUMP_DIR=… performHeapDump('manual')`
writes both a valid `.heapsnapshot` and a `.diagnostics.json`
containing detached-contexts, active-handles, smaps_rollup.
## Env knobs
- `HERMES_HEAPDUMP_DIR` — override snapshot output dir
- `HERMES_HEAPDUMP_ON_START=1` — dump once at boot
- existing `NODE_OPTIONS` is respected and appended, not replaced
VSCode panel-drag fires 20+ SIGWINCHes/sec, each previously triggering an unthrottled `terminal.resize` gateway RPC and a full transcript re-virtualization with stale per-row height cache. ## Changes ### gateway RPC debounce (ui-tui/src/app/useMainApp.ts) - `terminal.resize` RPC now trailing-debounced at 100 ms. React `cols` state stays synchronous (needed for Yoga / in-process rendering), only the round-trip to Python coalesces. Prevents gateway flood during panel-drag / tmux-pane-resize. ### column-aware useVirtualHistory (ui-tui/src/hooks/useVirtualHistory.ts) - New required `columns` param, plumbed through from useMainApp. - On column change: scale every cached row height by `oldCols/newCols` (Math.max 1, Math.round) instead of clearing. Clearing forces a pessimistic back-walk that mounts ~190 rows at once (viewport + 2x overscan at 1-row estimate), each a fresh marked.lexer + syntax highlight ≈ 3 ms — ~600 ms React commit block. Scaled heights keep the back-walk tight. - `freezeRenders=2`: reuse pre-resize mount range for 2 renders so already-mounted MessageRows keep their warm useMemo results. Without this the first post-resize render would unmount + remount most rows (pessimistic coverage) = visible flash + 150 ms+ freeze. - `skipMeasurement` flag: first post-resize useLayoutEffect would read PRE-resize Yoga heights (Yoga's stored values are still from the frame before this render's calculateLayout with new width) and poison the scaled cache. Skip the measurement loop for that one render; next render's Yoga is correct. ## Validation - tsc `--noEmit` clean - eslint clean on touched files - `vitest run`: 15 files / 102 tests passing The renderer-level resize patterns (sync-dim-capture + microtask- coalesced React commit, atomic BSU/ESU erase-before-paint, mouse- tracking reassert) already live in hermes-ink's own `handleResize`; this patch adds the matching app-layer hygiene.
KISS/DRY sweep — drops ~90 LOC with no behavior change. - circularBuffer: drop unused pushAll/toArray/size; fold toArray into drain - gracefulExit: inline Cleanup type + failsafe const; signal→code as a record instead of nested ternary; drop dead .catch on Promise.allSettled; drop unused forceExit - memory: inline heapDumpRoot() + writeSnapshot() (single-use); collapse the two fd/smaps try/catch blocks behind one `swallow` helper; build potentialLeaks functionally (array+filter) instead of imperative push-chain; UNITS at file bottom - memoryMonitor: inline DEFAULTS; drop unused onSnapshot; collapse dumpedHigh/dumpedCritical bools to a single Set; single callback dispatch line instead of duplicated if-chains - entry.tsx: factor `dumpNotice` formatter (used twice by onHigh + onCritical) - useMainApp resize debounce: drop redundant `if (timer)` guards (clearTimeout(undefined) is a no-op); init as undefined not null - useVirtualHistory: trim wall-of-text comment to one-line intent; hoist `const n = items.length`; split comma-declared lets; remove the `;[start, end] = frozenRange` destructure in favor of direct Math.min clamps; hoist `hi` init in upperBound for consistency Validation: tsc clean (both configs), eslint clean on touched files, vitest 102/102, build produces shebang-preserved dist/entry.js, performHeapDump smoke-test still writes valid snapshot + diagnostics.
OutThisLife
force-pushed
the
bb/tui-node-oom-hardening
branch
from
April 20, 2026 23:59
a702470 to
82b9277
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the Hermes TUI against long-session memory growth (Node/V8 heap limits + GatewayClient leaks) and improves resize responsiveness by reducing gateway RPC churn and stabilizing virtualized transcript rendering during column changes.
Changes:
- Adds heapdump + memory diagnostics utilities (manual
/heapdump,/mem, plus automatic dumps/exit on high/critical memory). - Refactors
GatewayClientto reduce memory churn/leaks (bounded buffers, safer timeout handling, clears timers on rejection). - Improves resize performance (debounced
terminal.resizeRPC; column-aware virtual history with scaled cached heights + temporary render/measurement freezing).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| ui-tui/src/lib/memoryMonitor.ts | Adds periodic heap monitoring with auto-dump triggers. |
| ui-tui/src/lib/memory.ts | Implements heap snapshot + diagnostics capture and byte formatting helpers. |
| ui-tui/src/lib/gracefulExit.ts | Adds signal handling + cleanup + failsafe exit wiring. |
| ui-tui/src/lib/circularBuffer.ts | Introduces fixed-capacity buffer to prevent unbounded log/event growth. |
| ui-tui/src/hooks/useVirtualHistory.ts | Makes virtualization column-aware; scales cached row heights across resizes to avoid heavy remount/measurement. |
| ui-tui/src/gatewayClient.ts | Uses circular buffers for logs/events; refactors RPC timeout lifecycle to reduce leaks. |
| ui-tui/src/entry.tsx | Wires memory monitor + graceful exit; adds memory/OOM resilience; updates shebang with heap flags. |
| ui-tui/src/app/useMainApp.ts | Debounces terminal.resize RPC and plumbs column count into virtualization hook. |
| ui-tui/src/app/slash/registry.ts | Registers new debug slash commands. |
| ui-tui/src/app/slash/commands/debug.ts | Adds /heapdump and /mem commands for live diagnostics. |
| hermes_cli/main.py | Appends Node heap/GC options when launching the TUI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
4 tasks
3 tasks
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
…om-hardening fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf
Luminet2023
pushed a commit
to Luminet2023/hermes-agent
that referenced
this pull request
May 1, 2026
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
Luminet2023
pushed a commit
to Luminet2023/hermes-agent
that referenced
this pull request
May 1, 2026
…om-hardening fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…om-hardening fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…om-hardening fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
Six small fixes, all valid review feedback: - gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a stable reference — no per-request bind allocation (the whole point of the original refactor). - memory: growth rate was lifetime average of rss/uptime, which reports phantom growth for stable processes. Now computed as delta since a module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at steady-state, non-zero after an allocation. - hermes_cli: NODE_OPTIONS merge is now token-aware — respects a user-supplied --max-old-space-size (don't downgrade a deliberate 16GB setting) and avoids duplicating --expose-gc. - useVirtualHistory: if items shrink past the frozen range's start mid-freeze (/clear, compaction), drop the freeze and fall through to the normal range calc instead of collapsing to an empty mount. - circularBuffer: throw on non-positive capacity instead of silently producing NaN indices. - debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override instead of hardcoding the default path. Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke test confirms baseline=0 → post-alloc>0.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…om-hardening fix(tui): harden against Node V8 OOM + GatewayClient leaks + resize perf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two classes of TUI pain folded into one branch:
splice'd event/log arrays bled memory during bursts.terminal.resizegateway RPC and re-runninguseVirtualHistorywith a now-wrong height cache → 200–600 ms React commit block + visible black flash.Defense-in-depth for both, plus a
/heapdump+/memfor when things do go sideways.1 — Memory hardening
Heap budget — never crash on OOM again
hermes_cli/main.py_launch_tui()injectsNODE_OPTIONS=--max-old-space-size=8192 --expose-gc, appending (not clobbering) any user-providedNODE_OPTIONS. Covers bothnode dist/entry.jsandtsx src/entry.tsxpaths.ui-tui/src/entry.tsxshebang upgraded to#!/usr/bin/env -S node --max-old-space-size=8192 --expose-gcas a fallback for direct invocation.GatewayClientleak-proofingsetMaxListeners(0)in the constructor — silences spurious warnings caused by legitimate multi-listener hook patterns.CircularBufferforlogsandbufferedEvents— O(1) fixed-capacity push; kills the hot-patharray.splice(0, n)copy churn under bursty stderr.method/params/resolve/rejectfor the full 120 s request timeout:setTimeout(this.onTimeout.bind(this), REQUEST_TIMEOUT_MS, id)— the bound handler only closes overthisandid.Pendingrecord owns its timeout handle,.unref()'d so stuck requests don't keep the event loop alive.rejectPending()now clears timers (previously leaked the timer itself on gateway exit/error).Memory diagnostics (new)
ui-tui/src/lib/memory.tsperformHeapDump()+captureMemoryDiagnostics()— writes both a V8.heapsnapshotand a.diagnostics.jsonsidecar to~/.hermes/heapdumps/(override viaHERMES_HEAPDUMP_DIR). Diagnostics written before the snapshot so we still get useful info if snapshot serialization crashes on large heaps.process._getActiveHandles()/_getActiveRequests(), Linux/proc/self/fdcount +/proc/self/smaps_rollup, heap growth rate (MB/hr), and auto-classifies likely leak sources.ui-tui/src/lib/memoryMonitor.ts— 10 s polling loop:heapUsed ≥ 1.5 GB→ auto heap dump (triggerauto-high)heapUsed ≥ 2.5 GB→ final dump +process.exit(137)so the user can restart cleanly before V8 fatal-OOMs.unref()'d; never holds the process open.Graceful exit (new)
ui-tui/src/lib/gracefulExit.tssetTimeoutthat hard-exits if cleanup hangs (WSL / tmux SIGHUP flakiness gets caught here).uncaughtException/unhandledRejectionlog to stderr instead of crashing — a transient TUI render error should not kill an in-flight agent turn.Slash commands (new)
/heapdump— manual snapshot + diagnostics with paths echoed into the transcript./mem— live heap / heapTotal / external / arrayBuffers / rss / uptime panel.Utility (new)
ui-tui/src/lib/circularBuffer.ts— small fixed-capacity ring (push/tail(n)/drain()/clear()). Replaces ad-hocarray.splice(0, len - MAX)pattern.2 — Resize perf
terminal.resizeRPC debounce —ui-tui/src/app/useMainApp.tscolsstate stays sync (Yoga / in-process rendering needs it); only the round-trip to Python coalesces. Stops gateway flood during panel-drag / tmux-pane-resize.Column-aware
useVirtualHistory—ui-tui/src/hooks/useVirtualHistory.tsNew required
columnsparam, plumbed through fromuseMainApp. Three coordinated fixes:oldCols/newColson column change. Clearing instead would force a pessimistic back-walk mounting ~190 rows at once — each a freshmarked.lexer+ syntax highlight ≈ 3 ms → ~600 ms React commit block. Scaled heights keep the back-walk tight.freezeRenders=2: reuse pre-resize mount range for 2 renders so already-mounted MessageRows keep their warmuseMemoresults. Without this the first post-resize render unmounts + remounts most rows = visible flash + 150 ms+ freeze.skipMeasurementflag: first post-resizeuseLayoutEffectwould read pre-resize Yoga heights (Yoga's stored values are still from the frame before this render'scalculateLayoutwith the new width) and poison the scaled cache. Skip the measurement loop for that one render; next render's Yoga is correct.The renderer-level patterns (sync-dim-capture + microtask-coalesced React commit, atomic BSU/ESU erase-before-paint, mouse-tracking reassert, iTerm2
?1049havoidance) already live in hermes-ink's ownhandleResize; this patch adds the matching app-layer hygiene.Env knobs
NODE_OPTIONSHERMES_HEAPDUMP_DIRHERMES_HEAPDUMP_ON_START=1Test plan
tsc --noEmit -p tsconfig.jsoncleanvitest run— 15 files, 102 tests passingeslintclean on all new / touched filestsc -p tsconfig.build.jsonproduces executabledist/entry.jswith the shebang preservedHERMES_HEAPDUMP_DIR=/tmp/hh performHeapDump('manual')→ valid.heapsnapshot+.diagnostics.jsonwithdetachedContexts,activeHandles,smapsRollup./membelow 1.5 GB, (c)/heapdumpproduces a useful snapshot if it does leak.Ctrl+C/ terminal-close still exits cleanly within the 4 s failsafe.