fix(dashboard): raise fd soft limit + replace iterdir with scandir to stop fd leak

[smfworks]

Summary

Long-running hermes dashboard exhausts its file descriptor soft limit and every os.listdir / open starts raising OSError [Errno 24] Too many open files (#81547). The dashboard process stays alive (uvicorn swallows the exception) but every API endpoint that touches Path.iterdir() or os.listdir silently fails — remote clients see connection timeouts.

Two root causes, two fixes:

1. Raise RLIMIT_NOFILE soft limit on startup

macOS defaults to 256 open files for LaunchAgent-managed processes. The dashboard opens 3 fds (db + wal + shm) per SessionDB per request across all profiles, and the sidebar polls every few seconds. After a few days the soft limit exhausts.

_raise_fd_soft_limit() runs before uvicorn binds and raises the soft limit to the hard limit (or minimum 4096). This matches the reporter's workaround (ulimit -n in the plist) but runs in-process so it works for all launch methods. No-op on Windows (no resource module) and when the limit is already adequate.

2. Replace bare Path.iterdir() with context-managed os.scandir()

Path.iterdir() returns a generator that holds an open directory fd until fully consumed. If an exception interrupts iteration, the fd leaks. On the sidebar poll path (every few seconds), this accumulates over days.

Fixed four hot paths:

• _fallback_profile_dicts — profile listing (polled on every sidebar refresh)
• File manager /api/fs/managed — directory listing
• Checkpoint listing — size calculation
• Plugin discovery — manifest scanning

os.scandir() is an explicit context manager (with os.scandir(...) as scan:) that guarantees the directory fd is closed on exit, following the same idiom already used in /api/fs/list.

Closes #81547.

Changes

File	Change
hermes_cli/web_server.py	+_raise_fd_soft_limit() helper (47 lines) — raises RLIMIT_NOFILE soft to hard/min 4096 before uvicorn binds. Called at top of start_server().
hermes_cli/web_server.py	4× Path.iterdir() → os.scandir() context manager in _fallback_profile_dicts, file manager list, checkpoint listing, plugin discovery. DirEntry → Path conversion at each call site.
tests/hermes_cli/test_dashboard_fd_leak_81547.py	9 new tests: import, Windows no-op, raise-when-low, noop-when-high, setrlimit-failure, and 4 source-level checks that iterdir is gone from each hot path.

Design Notes

• Stopgap + structural: Raising the fd limit prevents the OSError from masking the underlying bug. Replacing iterdir with scandir fixes the actual fd leak. Both are needed — the limit raise buys time, the scandir fix stops the leak.
• Cross-platform: _raise_fd_soft_limit silently no-ops on Windows (no resource module). os.scandir works on all platforms.
• No new env vars: The limit raise is automatic, not configurable. 4096 is a sane minimum that covers the dashboard's workload without being excessive. Users who need more can raise the hard limit in their launchd plist.

Test Plan

• tests/hermes_cli/test_dashboard_fd_leak_81547.py — 6 passed, 3 skipped (resource-module tests skip on Windows)
• No remaining Path.iterdir() in the four hot paths (source-level assertion)
• _raise_fd_soft_limit called before uvicorn binds in start_server()
• Existing os.scandir pattern in /api/fs/list unchanged — our changes follow the same idiom

Adversarial 6-Check — PASS

1. Diff integrity: Every change traces to dashboard fd leak: OSError [Errno 24] Too many open files after several days on macOS #81547 ✅
2. Blast radius: Only web_server.py + new test file ✅
3. Call ordering: _raise_fd_soft_limit() runs before import uvicorn in start_server() ✅
4. Type safety: All DirEntry → Path conversions present at each call site ✅
5. No regression: Existing /api/fs/list scandir pattern unchanged; all converted paths preserve sort order and filtering ✅
6. No remaining iterdir: Zero bare Path.iterdir() calls in the four fixed hot paths ✅

[teknium1]

Merged via PR #83542 — your commit(s) were cherry-picked onto current main with your authorship preserved in git log (rebase merge). Thank you for the contribution!

This follow-up PR completed the EMFILE hardening cluster after #83406: restart-path gateway orphan reap, Desktop-managed gateway termination on serve shutdown, SSH-spawn ulimit raise, and the dashboard iterdir→scandir fd-leak fixes. Everything was live-tested end-to-end on a real serve backend before merge, including a hostile unreadable-profile-dir fixture that surfaced (and fixed) a pre-existing /api/profiles 500 along the way.