fix(gateway): raise RLIMIT_NOFILE soft limit at startup (#30230) by briandevans · Pull Request #30234 · NousResearch/hermes-agent

briandevans · 2026-05-22T05:18:15Z

What does this PR do?

macOS ships a default RLIMIT_NOFILE soft limit of 256. Hermes gateways with multiple MCP subprocesses + per-profile instances routinely exceed this and crash session save / kanban dispatch with OSError: [Errno 24] Too many open files: '.sessions_*.tmp'.

This raises the soft limit toward 4096 (capped at the hard limit) at module init, right next to _ensure_ssl_certs(). Windows and sandboxed environments gracefully no-op. It is the smallest mitigation that addresses the root cap; the per-shutdown auxiliary-client reap landed in #14210 only delays the symptom, it doesn't fix the cap.

Related Issue

Fixes #30230

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
🔒 Security fix
📝 Documentation update
✅ Tests (adding or improving test coverage)
♻️ Refactor (no behavior change)
🎯 New skill (bundled or hub)

Changes Made

gateway/run.py — add _raise_fd_soft_limit(min_soft=4096) helper and call it at module init right after _ensure_ssl_certs(). Bumps toward min(min_soft, hard); no-ops when soft is already above target; swallows OSError/ValueError so sandboxed envs (and Windows, which has no resource module) degrade cleanly.
tests/gateway/test_fd_soft_limit.py — 8 cases covering: bump-from-256 with infinite hard, cap-at-hard when hard < target, no-op when soft already high, no-op when soft == hard == below target, getrlimit failure swallowed, setrlimit failure swallowed, custom min_soft override, and a source-anchor test that pins the production keywords so the replica can't silently drift.

How to Test

uv run --with pytest --with pytest-xdist --with pytest-asyncio --with pytest-timeout python3 -m pytest tests/gateway/test_fd_soft_limit.py tests/gateway/test_ssl_certs.py tests/gateway/test_runner_startup_failures.py tests/gateway/test_allowlist_startup_check.py -v
Expected: 24 passed.
Manual: on a macOS shell, python3 -c "import resource; r=resource.getrlimit(resource.RLIMIT_NOFILE); print(r); import gateway.run; print(resource.getrlimit(resource.RLIMIT_NOFILE))" shows the soft limit rising from 256 → 4096 after import.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
I searched for existing PRs to make sure this isn't a duplicate
My PR contains only changes related to this fix/feature (no unrelated commits)
I've run focused tests for the touched code and all pass
I've added tests for my changes (required for bug fixes, strongly encouraged for features)
I've tested on my platform: macOS 15.x

Documentation & Housekeeping

I've updated relevant documentation (README, docs/, docstrings) — N/A (docstring on _raise_fd_soft_limit is the relevant surface)
I've updated cli-config.yaml.example if I added/changed config keys — N/A
I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
I've considered cross-platform impact (Windows, macOS) per the compatibility guide — Windows has no resource module; helper catches ImportError and returns early. Sandboxed Linux containers that forbid setrlimit (seccomp, restrictive cgroups) catch OSError/ValueError.
I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Before (macOS, fresh gateway process):

$ launchctl limit maxfiles
maxfiles    256            unlimited
$ python3 -c "import gateway.run, resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))"
(256, 9223372036854775807)

After:

$ python3 -c "import gateway.run, resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))"
(4096, 9223372036854775807)

Audited siblings: there is no _ensure_ssl_certs-class helper outside gateway/run.py that needs the same bump — gateway/run.py is the gateway main entry, and the CLI / cron paths spawn fewer simultaneous fd-heavy subprocesses (no MCP fan-out). No widening needed.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a Unix-only file descriptor (RLIMIT_NOFILE) soft-limit bump during gateway startup, along with focused tests that validate the behavior and try to prevent drift from the production implementation.

Changes:

Introduce _raise_fd_soft_limit() in gateway/run.py and invoke it during module initialization.
Add a new test suite that validates limit bumping, capping behavior, and error swallowing.
Add a “drift guard” test that pins key strings in gateway/run.py to keep the test replica aligned.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.

File	Description
`gateway/run.py`	Adds and calls `_raise_fd_soft_limit()` to reduce EMFILE risk by bumping RLIMIT_NOFILE on Unix.
`tests/gateway/test_fd_soft_limit.py`	Adds tests for RLIMIT bump logic plus a lightweight production-source pin to reduce drift.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+class TestRaiseFdSoftLimit:
+    def test_bumps_from_256_to_4096_when_hard_is_infinity(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_caps_at_hard_when_hard_below_target(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_noop_when_soft_already_high(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_noop_when_soft_equals_hard_below_min(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_swallows_getrlimit_error(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_swallows_setrlimit_error(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+
+    def test_custom_min_soft_threshold(self):
+        fn = _load_raise_fd_soft_limit()
+        import resource


+def _load_raise_fd_soft_limit():
+    """Replicate the helper in an isolated module.
+
+    gateway/run.py has heavy imports; tests/gateway/test_ssl_certs.py uses
+    the same pattern.  The body below must stay in sync with the production
+    function in gateway/run.py.
+    """
+    code = textwrap.dedent("""\
+    def _raise_fd_soft_limit(min_soft=4096):
+        try:
+            import resource
+        except ImportError:
+            return
+        try:
+            soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+        except (OSError, ValueError):
+            return
+        if soft >= min_soft:
+            return
+        target = min_soft if hard == resource.RLIM_INFINITY else min(min_soft, hard)
+        if target <= soft:
+            return
+        try:
+            resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))
+        except (OSError, ValueError):
+            pass
+    """)


+        text = src.read_text()
+        assert "def _raise_fd_soft_limit(" in text
+        assert "RLIMIT_NOFILE" in text
+        assert "RLIM_INFINITY" in text
+        # Helper is wired into module init right after _ensure_ssl_certs().
+        assert "_raise_fd_soft_limit()" in text


@@ -513,6 +513,41 @@ def _ensure_ssl_certs() -> None:
            os.environ["SSL_CERT_FILE"] = candidate
            return



talwayh1 · 2026-05-25T05:30:27Z

CI Flake Found & Fix Available

CI self-heal detected a flake in the test suite on run 26384541008:

Failure

FAILED tests/tools/test_local_interrupt_cleanup.py::test_wait_for_process_kills_subprocess_on_keyboardinterrupt
Failed: Timeout (>30.0s) from pytest-timeout.

Root Cause

The test internally budgets ~50s (5s subprocess discovery + 15s worker-thread join + 30s process-group-exit poll) but the suite default --timeout=30 kills it before _wait_for_pgid_exit even gets a meaningful poll window. Under heavy xdist load (6-shard CI), the cleanup chain (SIGTERM → reap → SIGKILL → reap) can lag enough that the 30s global cap fires inside the polling function.

Fix

Add @pytest.mark.timeout(90) to give the test its full budget plus headroom for CI scheduling jitter.

Commit: cd06767b9 on talwayh1/hermes-agent:fix/gateway-raise-fd-soft-limit-30230

# To apply:
git fetch https://github.com/talwayh1/hermes-agent.git fix/gateway-raise-fd-soft-limit-30230
git cherry-pick cd06767b9

…#30230) macOS ships a default RLIMIT_NOFILE soft limit of 256. Hermes gateways with multiple MCP subprocesses + per-profile instances routinely exceed this and crash session save / kanban dispatch with OSError [Errno 24]. Bump the soft limit toward 4096 (capped at the hard limit) at module init alongside _ensure_ssl_certs. Windows / sandboxed environments gracefully no-op. This is the smallest mitigation that addresses the root cap; it complements the per-shutdown auxiliary-client reap landed in NousResearch#14210, which only delays the symptom. Tests pin the in-test replica against the production source so the helper can't silently drift.

briandevans · 2026-06-03T04:13:31Z

Closing to focus the queue on security/file-safety work where civilian merges are landing. Happy to reopen if maintainers want this picked up.

Copilot AI review requested due to automatic review settings May 22, 2026 05:18

Copilot AI reviewed May 22, 2026

View reviewed changes

alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels May 22, 2026

briandevans force-pushed the fix/gateway-raise-fd-soft-limit-30230 branch 2 times, most recently from 41b2d6a to b9e1a51 Compare May 25, 2026 05:15

briandevans force-pushed the fix/gateway-raise-fd-soft-limit-30230 branch from b9e1a51 to bde0826 Compare May 27, 2026 15:16

briandevans force-pushed the fix/gateway-raise-fd-soft-limit-30230 branch from bde0826 to 3c085f1 Compare May 29, 2026 15:13

briandevans closed this Jun 3, 2026

This was referenced Aug 3, 2026

[Bug]: Desktop SSH serve inherits macOS 256-FD limit with no YAML override #77573

Closed

fix(runtime): make nofile soft limit configurable #77587

Closed

Navigation Menu

Uh oh!

fix(gateway): raise RLIMIT_NOFILE soft limit at startup (#30230) - #30234

fix(gateway): raise RLIMIT_NOFILE soft limit at startup (#30230)#30234
briandevans wants to merge 1 commit into
NousResearch:mainfrom
briandevans:fix/gateway-raise-fd-soft-limit-30230

briandevans commented May 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

talwayh1 commented May 25, 2026

Uh oh!

briandevans commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -513,6 +513,41 @@ def _ensure_ssl_certs() -> None:
		os.environ["SSL_CERT_FILE"] = candidate
		return

Uh oh!

Conversation

briandevans commented May 22, 2026

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Checklist

Code

Documentation & Housekeeping

Screenshots / Logs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

talwayh1 commented May 25, 2026

CI Flake Found & Fix Available

Failure

Root Cause

Fix

Uh oh!

briandevans commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants