fix(gateway): fix fd reclaim bug that can wedge long-running gateways - #41243
fix(gateway): fix fd reclaim bug that can wedge long-running gateways#41243ProgramCaiCai wants to merge 3 commits into
Conversation
|
Positive verification — clean fix with strong test coverage. The owner-chain tracking ( The The FD-count regression test ( The No issues found. |
895dbf3 to
605eb63
Compare
(cherry picked from commit 895dbf3e3cd67486300408174899eb9fb5bdf502)
605eb63 to
6fc66f5
Compare
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the detailed FD-lifecycle investigation. The remaining auxiliary-cache and launchd work is still relevant on current main: FIFO eviction in agent/auxiliary_client.py:5842-5845 does not call close(), and hermes_cli/gateway.py:3962-3970 has no launchd NumberOfFiles limits.
Problems
- The Feishu hunk is already implemented in
plugins/platforms/feishu/adapter.py:1773-1815, with regression coverage attests/gateway/test_feishu.py:277-346(77700a0ec). - The Telegram hunk conflicts with the current intentional fallback boundary:
plugins/platforms/telegram/adapter.py:3579-3580returns retryablesend_path_degraded, andtests/gateway/test_telegram_send_path_health.py:53-64specifies that callers perform standalone delivery. - The changed Feishu/Telegram paths were migrated to plugins by
560010547, so this needs a targeted salvage rather than a clean cherry-pick.
Suggested changes
- Port and revalidate only the auxiliary-cache and launchd pieces against current main; omit the duplicate Feishu change and separately justify any Telegram fallback redesign.
Automated hermes-sweeper review.
| else: # "first" (default) | ||
| return chunk_index == 0 | ||
|
|
||
| async def _send_via_standalone_fallback( |
There was a problem hiding this comment.
This is an unrelated fallback-policy change. Current main intentionally returns retryable send_path_degraded so the caller owns standalone delivery (plugins/platforms/telegram/adapter.py:3579-3580; tests/gateway/test_telegram_send_path_health.py:53-64). Please split it from the FD-reclaim salvage or revalidate that contract end-to-end.
Why
RLIMIT_NOFILE, the gateway can enter anEMFILEfailure mode where it is still alive but can no longer open the sockets/files it needs to serve traffic.RLIMIT_NOFILEof 256 unless the generated plist declaresNumberOfFiles.Root cause
This PR fixes the two confirmed fd-lifecycle gaps behind that long-running failure mode:
agent/auxiliary_client.py: cache eviction/replacement/shutdown/stale-loop cleanup was not consistently closing the real owner client that holds the underlying httpx poolgateway/platforms/feishu.py:disconnect()stopped the websocket loop without explicitly calling the SDK websocket client_disconnect(), so teardown could leave the underlying socket open longer than intendedHow
disconnect()before stopping the websocket thread loopSoftResourceLimits/HardResourceLimits -> NumberOfFiles = 4096/8192, reducing blast radius for service-managed gatewaysWhy this is the complete fix path
Related open PRs each address only part of the problem:
Those changes reduce blast radius or fix adjacent leaks, but they do not close the confirmed reclaim holes that let long-running gateways accumulate fds until they wedge.
Tests
/Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/run_agent/test_async_httpx_del_neuter.py -q/Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/agent/test_auxiliary_client.py -q/Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/gateway/test_feishu.py -q/Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/gateway/test_safe_adapter_disconnect.py -q/Users/programcaicai/.hermes/hermes-agent/venv/bin/python -m pytest tests/hermes_cli/test_gateway_service.py -q -k launchdLinked issue
Closes #41242