fix(runtime): make nofile soft limit configurable

[100yenadmin]

What does this PR do?

Adds one shared, configurable RLIMIT_NOFILE soft-limit floor for Hermes' long-running gateway and dashboard/serve processes.

This closes the macOS Desktop SSH gap where hermes serve --isolated inherits a soft limit of 256. Under file-descriptor pressure that produced [Errno 24] Too many open files and cascading auth/session/provider failures even with only a few visible agents.

Configuration

runtime:
  nofile_soft_limit: 4096

• default: 4096
• finite hard limit: target is clamped
• existing higher/unlimited soft limit: never lowered
• 0, false, or null: disable the adjustment
• Windows, malformed config, and denied setrlimit: safe no-op; startup continues

The implementation uses the canonical profile-aware load_config_readonly() path, including managed-scope overlays. It does not add an environment-variable side channel.

Startup surfaces

• gateway.run.start_gateway() — applied before gateway initialization
• hermes_cli.main.cmd_dashboard() — applied after --status/--stop early exits and before web-server imports/state; covers Desktop SSH hermes serve --isolated

Prior art and scope

• Builds on the sound self-raise idea in closed PR fix(gateway): raise RLIMIT_NOFILE soft limit at startup (#30230) #30234 rather than duplicating it in a shell-only launcher.
• Complements [codex] Fix macOS gateway fd exhaustion paths #50494, which configures launchd-managed gateways but does not affect SSH-spawned Desktop backends.
• Complements fix(state): reap finished-thread WAL read connections at runtime (#75269) #75322, which fixes retained SQLite readers; this PR adds process-capacity defense in depth and does not replace cleanup.

Contributor

• Name: Eva / 100yenadmin
• GitHub: @100yenadmin
• Role: External contributor

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change
• Documentation update
• Performance improvement
• Test coverage improvement
• Refactoring (no functional changes)
• Other

Related issue

Fixes #77573
Related: #30230, #30234, #50494, #75269, #75322

Testing performed

PYTHONPATH=. python -m pytest -q \
  tests/test_resource_limits.py \
  tests/gateway/test_runner_startup_failures.py \
  tests/hermes_cli/test_dashboard_web_dist_validation.py \
  tests/hermes_cli/test_serve_command.py
# 33 passed

scripts/run_tests.sh \
  tests/test_resource_limits.py \
  tests/gateway/test_runner_startup_failures.py \
  tests/hermes_cli/test_dashboard_web_dist_validation.py \
  tests/hermes_cli/test_serve_command.py -q
# 33 passed under the repository's clean-env parallel runner

scripts/run_tests.sh \
  tests/hermes_cli/test_config.py \
  tests/hermes_cli/test_config_loader_e2e.py \
  tests/hermes_cli/test_config_validation.py \
  tests/gateway/test_config.py -q
# 139 passed under the repository's clean-env parallel runner

ruff check \
  hermes_cli/resource_limits.py \
  tests/test_resource_limits.py \
  gateway/run.py \
  hermes_cli/main.py \
  hermes_cli/config_defaults.py
# All checks passed!

python -m compileall -q hermes_cli/resource_limits.py tests/test_resource_limits.py
git diff --check

Real macOS subprocess probes:

• default: (256, unlimited) → (4096, unlimited)
• custom 512: (256, unlimited) → (512, unlimited)
• disabled 0: remains (256, unlimited)

Real candidate CLI on a throwaway profile:

hermes config set runtime.nofile_soft_limit 512
hermes config get runtime.nofile_soft_limit
# 512

The test suite directly covers:

• canonical config loading
• finite hard-limit clamping
• higher and unlimited soft-limit preservation
• 0 / false / null opt-out
• malformed values
• unsupported platforms
• finite soft limit with an infinite hard limit (the macOS shape)
• fresh-process import without the POSIX resource module
• getrlimit failure
• denied setrlimit
• gateway ordering
• serve/dashboard ordering
• named-profile dashboard reroute policy isolation
• --status / --stop non-mutation

Backward compatibility

No migration is required. Existing configs receive the default through Hermes' normal deep-merge behavior. The operation is best-effort and never lowers an existing limit. Users can opt out with runtime.nofile_soft_limit: 0 (also false or null).

Checklist

• I have read the contribution guidelines
• My code follows the project's style guidelines
• I have performed a self-review of my own code
• I have added tests that prove the fix is effective or that the feature works
• New and existing targeted tests pass locally
• Full repository test suite passes locally (not run; targeted startup/config suites passed)
• I have added/updated documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

[leonphull]

Production validation + cherry-pick notice

We reproduced the macOS soft-256 EMFILE path on a multi-profile Desktop host and needed this floor in the field.

Cherry-picked into #78873 (author commits preserved: 89d722cdc, bfffb7ec7) alongside:

• parent-death serve watchdog (fix(desktop): reap orphaned serve backends via parent-death watchdog + group-kill #73066)
• startup reap for already orphaned Desktop-local serves (new)

If maintainers prefer this PR to land first on its own, that’s ideal — #78873 can then drop the cherry-picks and keep only the startup-reap commit. Happy to rebase either way.

Related incident: #78872 / #77573

[teknium1]

Merged via PR #83406 — your commits were cherry-picked onto current main with your authorship preserved in git log (rebase merge). Thank you for the contribution!

The omnibus combined the WAL read-connection pool (#76700), the configurable nofile floor (#77587), the parent-death watchdog (#73066), the Desktop-boot orphan serve reap incl. the backend.lock.json spare-guard (#78873), and the orphan gateway reap (#78312). All pieces were live-tested end-to-end: real orphaned processes reaped on actual Desktop boot with a lock-owned backend surviving, 151→9 connections under 150 reader threads, and zero fd growth over 800 requests against a live serve.