fix(gateway): stop macOS gateway EMFILE — raise launchd fd limit + fix Telegram pool leak - #39336
Open
gumclaw wants to merge 2 commits into
Open
fix(gateway): stop macOS gateway EMFILE — raise launchd fd limit + fix Telegram pool leak#39336gumclaw wants to merge 2 commits into
gumclaw wants to merge 2 commits into
Conversation
launchd agents inherit macOS's default soft open-file limit of 256.
A long-lived gateway exhausts this — a large ~/.hermes/sessions/ dir
plus accumulated async httpx transports hit EMFILE ('[Errno 24] Too
many open files') on session .tmp writes, breaking the session and
forcing a /reset. The kernel ceiling (kern.maxfilesperproc) is far
higher, so 256 was the only real constraint.
Add SoftResourceLimits/HardResourceLimits NumberOfFiles (default
65536, overridable via HERMES_GATEWAY_MAX_FILES) to the generated
launchd plist. Because the plist is reconciled against this template
on every gateway start/restart/setup, the limit now persists instead
of reverting to 256.
Refs NousResearch#14210
…OSE_WAIT leak
TelegramFallbackTransport builds its inner httpx.AsyncHTTPTransport
instances with no limits, so they silently use httpx's defaults
(max_connections=100, max_keepalive=20, keepalive_expiry=5s). PTB's
connection_pool_size=512 / pool_timeout settings only configure the
OUTER httpx.AsyncClient, and httpx ignores the outer client's limits
once a custom transport is supplied — so on the fallback-IP path the
effective pool was 100, not 512.
Under a long-lived gateway this manifests as:
- 'Pool timeout: All connections in the connection pool are
occupied. Request was *not* sent to Telegram.' send failures, and
- hundreds of server-closed sockets stuck in CLOSE_WAIT (the 5s
keepalive churns connections faster than they're reused),
which combine with the 256 launchd fd cap to trigger EMFILE.
Inject explicit httpx.Limits into every inner transport (primary +
each fallback), defaulting to 512 connections / 512 keepalive / 20s
expiry, overridable via HERMES_TELEGRAM_HTTP_POOL_SIZE and
HERMES_TELEGRAM_HTTP_KEEPALIVE_EXPIRY. Caller-supplied limits are
respected. The non-fallback path already honored connection_pool_size
via PTB's default transport and is unaffected.
Refs NousResearch#14210
alt-glitch
added
type/bug
This was referenced Jun 7, 2026
Open
13 tasks
Contributor
|
Thanks for the detailed EMFILE investigation. The launchd portion remains relevant: current Problems
Suggested changes
Automated hermes-sweeper review. |
teknium1
added
sweeper:risk-message-delivery
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes
[Errno 24] Too many open files(EMFILE) on the macOS launchd gateway, which breaks session writes (.tmpfiles in~/.hermes/sessions/) and forces a/reset.Two compounding root causes:
RLIMIT_NOFILEof 256. A long-lived gateway with a large~/.hermes/sessions/dir plus accumulatedhttpxtransports exhausts it. The generated plist set no resource limit, and the plist is reconciled against the template on every start/restart/setup — so any manual edit was silently reverted to 256.TelegramFallbackTransportbuilds its innerhttpx.AsyncHTTPTransportinstances with no limits, so they fall back to httpx defaults (max_connections=100,max_keepalive=20,keepalive_expiry=5s). PTB'sconnection_pool_size=512/pool_timeoutonly configure the outerhttpx.AsyncClient, and httpx ignores the outer client's limits once a custom transport is supplied. On the fallback-IP path the effective pool was 100, producingPool timeout: All connections in the connection pool are occupiedsend failures and hundreds ofCLOSE_WAITsockets that combine with the 256 cap to trigger EMFILE.Related Issue
Refs #14210
Type of Change
Changes Made
SoftResourceLimits/HardResourceLimits→NumberOfFiles(default65536, overridable viaHERMES_GATEWAY_MAX_FILES) in the generated plist, so the limit persists across template reconciliation.TelegramFallbackTransport— Inject explicithttpx.Limitsinto every inner fallback transport (primary + each fallback): 512 conns / 512 keepalive / 20s expiry, overridable viaHERMES_TELEGRAM_HTTP_POOL_SIZEandHERMES_TELEGRAM_HTTP_KEEPALIVE_EXPIRY. Caller-supplied limits are respected. The non-fallback path already honoredconnection_pool_sizeand is unaffected.How to Test
Automated:
tests/hermes_cli/test_gateway_service.py— launchd plist sets the fd limit, honors theHERMES_GATEWAY_MAX_FILESenv override, and falls back to the default on invalid input.tests/gateway/test_telegram_network.py— inner fallback transports get the large pool, the env override applies, an invalid env value falls back, and explicitly-supplied limits are respected.Manual (reproduces the original EMFILE):
hermes gateway install). The inherited soft cap is 256 (launchctl limit maxfiles), and under sustained session writes + Telegram fallback traffic the gateway logs[Errno 24] Too many open files.SoftResourceLimits NumberOfFilesis65536in the generated plist;CLOSE_WAITsockets stay flat (lsof -p <gateway_pid> | grep -c CLOSE_WAIT) and the EMFILE no longer occurs.Result: targeted tests pass;
ruffclean.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — N/Acli-config.yaml.exampleif I added/changed config keys — N/A (env-var overrides only)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/AFor New Skills
N/A
Screenshots / Logs
Failing path (before):
OSError: [Errno 24] Too many open fileswriting session.tmpunder~/.hermes/sessions/, alongsidePool timeout: All connections in the connection pool are occupiedon the Telegram fallback path.Validation run:
Passed.