Skip to content

fix(gateway): fix fd reclaim bug that can wedge long-running gateways - #41243

Open
ProgramCaiCai wants to merge 3 commits into
NousResearch:mainfrom
ProgramCaiCai:fix/gateway-fd-reclaim-launchd-4096
Open

fix(gateway): fix fd reclaim bug that can wedge long-running gateways#41243
ProgramCaiCai wants to merge 3 commits into
NousResearch:mainfrom
ProgramCaiCai:fix/gateway-fd-reclaim-launchd-4096

Conversation

@ProgramCaiCai

Copy link
Copy Markdown
Contributor

Why

  • Long-running gateways were not deterministically reclaiming every file descriptor they opened during real work. After tasks finished, fd counts could drift upward instead of returning near baseline.
  • Once the process approaches RLIMIT_NOFILE, the gateway can enter an EMFILE failure mode where it is still alive but can no longer open the sockets/files it needs to serve traffic.
  • In that wedged state the service manager does not automatically heal the gateway, because the process has not cleanly exited. Operators are left with a manually restarted gateway.
  • macOS launchd makes this easier to hit because service-managed gateways stay pinned to a tiny default soft RLIMIT_NOFILE of 256 unless the generated plist declares NumberOfFiles.

Root cause

This PR fixes the two confirmed fd-lifecycle gaps behind that long-running failure mode:

  • agent/auxiliary_client.py: cache eviction/replacement/shutdown/stale-loop cleanup was not consistently closing the real owner client that holds the underlying httpx pool
  • gateway/platforms/feishu.py: disconnect() stopped the websocket loop without explicitly calling the SDK websocket client _disconnect(), so teardown could leave the underlying socket open longer than intended

How

  • rework auxiliary-client cache disposal around owner-aware reclaim so shared sync/async wrappers close the real underlying client exactly once when the last cache owner disappears
  • explicitly close the live Feishu websocket client during disconnect() before stopping the websocket thread loop
  • update the generated macOS launchd plist to declare SoftResourceLimits/HardResourceLimits -> NumberOfFiles = 4096/8192, reducing blast radius for service-managed gateways

Why this is the complete fix path

Related open PRs each address only part of the problem:

  • #36939 and #38086 only raise the launchd fd ceiling
  • #38787 and #41144 only touch part of auxiliary-client eviction, but do not cover owner-aware disposal across replacement, shutdown, stale-loop cleanup, and same-key race loser paths
  • #39336 raises launchd limits and fixes a Telegram-specific leak, but it does not close the confirmed auxiliary-client plus Feishu reclaim gaps fixed here

Those changes reduce blast radius or fix adjacent leaks, but they do not close the confirmed reclaim holes that let long-running gateways accumulate fds until they wedge.

Tests

  • /Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/run_agent/test_async_httpx_del_neuter.py -q
  • /Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/agent/test_auxiliary_client.py -q
  • /Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/gateway/test_feishu.py -q
  • /Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/gateway/test_safe_adapter_disconnect.py -q
  • /Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/hermes_cli/test_gateway_service.py -q -k launchd

Linked issue

Closes #41242

@liuhao1024

Copy link
Copy Markdown
Contributor

Positive verification — clean fix with strong test coverage.

The owner-chain tracking (_cached_client_owner / _explicit_real_client) correctly handles the MagicMock auto-attribute pitfall — vars(client) + __slots__ check is the right way to distinguish a real _real_client wrapper from a mock that auto-creates it on access.

The _dispose_removed_cached_clients guard against double-disposing when two evicted entries share the same underlying owner is a subtle correctness detail that would be easy to miss.

The FD-count regression test (test_repeated_fifo_eviction_does_not_accumulate_os_fds) is excellent — it validates the actual OS-level invariant rather than just checking Python-level close() calls, which makes it resistant to refactors that change the internal disposal mechanism.

The _close_websocket_client Feishu change (5s timeout on asyncio.wait_for) is a reasonable safety net — without it a hung _disconnect() coroutine could block the gateway shutdown indefinitely.

No issues found.

@daimon-nous daimon-nous Bot added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint platform/feishu Feishu / Lark adapter labels Jun 7, 2026
@ProgramCaiCai
ProgramCaiCai force-pushed the fix/gateway-fd-reclaim-launchd-4096 branch from 895dbf3 to 605eb63 Compare June 16, 2026 03:05
@ProgramCaiCai
ProgramCaiCai force-pushed the fix/gateway-fd-reclaim-launchd-4096 branch from 605eb63 to 6fc66f5 Compare June 16, 2026 03:20

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed FD-lifecycle investigation. The remaining auxiliary-cache and launchd work is still relevant on current main: FIFO eviction in agent/auxiliary_client.py:5842-5845 does not call close(), and hermes_cli/gateway.py:3962-3970 has no launchd NumberOfFiles limits.

Problems

  • The Feishu hunk is already implemented in plugins/platforms/feishu/adapter.py:1773-1815, with regression coverage at tests/gateway/test_feishu.py:277-346 (77700a0ec).
  • The Telegram hunk conflicts with the current intentional fallback boundary: plugins/platforms/telegram/adapter.py:3579-3580 returns retryable send_path_degraded, and tests/gateway/test_telegram_send_path_health.py:53-64 specifies that callers perform standalone delivery.
  • The changed Feishu/Telegram paths were migrated to plugins by 560010547, so this needs a targeted salvage rather than a clean cherry-pick.

Suggested changes

  • Port and revalidate only the auxiliary-cache and launchd pieces against current main; omit the duplicate Feishu change and separately justify any Telegram fallback redesign.

Automated hermes-sweeper review.

else: # "first" (default)
return chunk_index == 0

async def _send_via_standalone_fallback(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an unrelated fallback-policy change. Current main intentionally returns retryable send_path_degraded so the caller owns standalone delivery (plugins/platforms/telegram/adapter.py:3579-3580; tests/gateway/test_telegram_send_path_health.py:53-64). Please split it from the FD-reclaim salvage or revalidate that contract end-to-end.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/feishu Feishu / Lark adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(gateway): incomplete fd reclaim can wedge long-running gateways until manual restart

3 participants