Skip to content

fix(desktop-ssh): raise remote backend file limit - #82909

Closed
a-espinoza wants to merge 1 commit into
NousResearch:mainfrom
a-espinoza:fix/desktop-ssh-nofile-limit-upstream
Closed

a-espinoza wants to merge 1 commit into
NousResearch:mainfrom
a-espinoza:fix/desktop-ssh-nofile-limit-upstream

Conversation

@a-espinoza

Copy link
Copy Markdown
Contributor

Summary

  • raise the soft file-descriptor limit for SSH-launched Desktop backends before exec
  • keep startup portable by falling back to the host's existing limit if the raise is rejected
  • cover command ordering and the 65,536 limit with a regression test

Root cause

On macOS, a non-interactive sshd shell can inherit a soft nofile limit of 256 even when the hard limit is unlimited. A long-lived hermes serve --isolated Desktop backend can exceed that limit while serving profiles and tools, after which unrelated reads fail with errors such as:

OSError: [Errno 24] Too many open files: '~/.hermes/auth.json'

The credential file is not corrupt; it is simply the next file the exhausted process attempted to open.

Fix

The detached SSH child now runs:

ulimit -n 65536 2>/dev/null || true

before exec env HERMES_DESKTOP=1 .... This changes only the Desktop-owned backend process and preserves existing behavior on hosts that reject the requested limit.

Verification

  • npm exec --workspace apps/desktop -- vitest run --project electron electron/remote-lifecycle.test.ts — 67 passed
  • npm exec --workspace apps/desktop -- eslint electron/remote-lifecycle.ts electron/remote-lifecycle.test.ts — passed
  • npm run typecheck --workspace apps/desktop — passed
  • reproduced MacBook → Mac mini SSH soft limit: 256
  • deployed packaged macOS app raised the launch path to 65,536; fresh isolated backend started successfully and /api/status returned HTTP 200

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/desktop Electron desktop app (apps/desktop/*) backend/ssh SSH remote execution labels Aug 10, 2026

@leonphull leonphull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent production validation from the original MacBook → Mac mini failure topology:

  • Fresh non-interactive SSH before mitigation: soft RLIMIT_NOFILE=256, hard unlimited.
  • Active Desktop-owned isolated backend after ~11h: 228 numeric FDs (124 state.db*, 25 sockets, 27 pipes) and prior real EMFILE cascades in auth/skills/cron/sidebar paths.
  • This PR's exact targeted suite passes at 038a45374: 67/67 Vitest, scoped ESLint, and Desktop typecheck.
  • Sabotage check: keeping the new regression test while restoring the pre-fix remote-lifecycle.ts fails exactly at the missing ulimit; restoring the implementation passes.
  • Real macOS /bin/sh probe confirms the launcher command raises a 256 inheritance to 65,536.

One portability/correctness edge is worth fixing before merge: under macOS /bin/sh, bare ulimit -n 65536 changes both soft and hard limits, so an unlimited hard limit becomes 65,536 even though the PR describes a soft-only raise. On a host with finite hard=4096, the command fails and leaves soft=256 instead of using the available 4096.

Observed:

unlimited hard: soft=65536 hard=65536
finite hard=4096: soft=256 hard=4096

Please use ulimit -S -n and clamp the target to a numeric finite hard limit. This POSIX sh shape was verified on the affected macOS host:

h=$(ulimit -H -n 2>/dev/null || printf '')
case "$h" in
  unlimited) t=65536 ;;
  ''|*[!0-9]*) t=4096 ;;
  *) if test "$h" -lt 65536; then t="$h"; else t=65536; fi ;;
esac
ulimit -S -n "$t" 2>/dev/null || true

Verified results:

unlimited hard: soft=65536 hard=unlimited
finite hard=4096: soft=4096 hard=4096

With that adjustment, this is the right minimal fix for the Desktop SSH launcher path. The configurable in-process floor in #77587 remains useful defense-in-depth for non-Desktop serve/gateway startup paths.

@teknium1

Copy link
Copy Markdown
Collaborator

Merged via PR #83542 — your commit(s) were cherry-picked onto current main with your authorship preserved in git log (rebase merge). Thank you for the contribution!

This follow-up PR completed the EMFILE hardening cluster after #83406: restart-path gateway orphan reap, Desktop-managed gateway termination on serve shutdown, SSH-spawn ulimit raise, and the dashboard iterdir→scandir fd-leak fixes. Everything was live-tested end-to-end on a real serve backend before merge, including a hostile unreadable-profile-dir fixture that surfaced (and fixed) a pre-existing /api/profiles 500 along the way.

@teknium1 teknium1 closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/ssh SSH remote execution comp/desktop Electron desktop app (apps/desktop/*) P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants