fix: enable TCP keepalives to detect dead provider connections (#1032…

[decadez]

…4) (NousResearch#11277)

Re-land of NousResearch#10933, now guarded by the tests in NousResearch#11266.

When a provider drops a TCP connection mid-stream, the socket can enter CLOSE-WAIT and ''epoll_wait'' may never fire — no data or error signal arrives, so the httpx read timeout never triggers and the agent hangs indefinitely. The other defenses (''_force_close_tcp_sockets'', stale stream detector) all ride on the socket layer reporting the dead connection, which it never does without probes.

Inject ''SO_KEEPALIVE'' + ''TCP_KEEPIDLE''/''KEEPINTVL''/''KEEPCNT'' into the httpx transport. Kernel probes after 30s idle, retries every 10s, gives up after 3 → dead peer detected within ~60s instead of hanging forever. Platform-aware: ''TCP_KEEPIDLE'' on Linux, ''TCP_KEEPALIVE'' on macOS. Silent no-op on Windows or anywhere the socket options aren't available.

The original land (NousResearch#10933) mutated ''client_kwargs'' in place when it injected the ''httpx.Client''. Since callers pass ''self._client_kwargs'' by reference, the injected client leaked into the instance state. After the first request, the OpenAI SDK closed its ''http_client'' — including the injected one. The next ''_create_openai_client'' call re-read the now-closed ''httpx.Client'' from ''self._client_kwargs'' and every subsequent chat raised ''APIConnectionError'' with cause ''RuntimeError: Cannot send a request, as the client has been closed'' (AlexKucera's Discord report, 2026-04-16).

The defensive ''client_kwargs = dict(client_kwargs)'' copy already on main (taeuk178's NousResearch#10978) means this injection only lands in the per-call local copy. Each ''_create_openai_client'' invocation gets its OWN fresh ''httpx.Client'' whose lifetime is tied to the paired ''OpenAI'' client. When that ''OpenAI'' client is closed (rebuild, teardown, credential rotation), its ''httpx.Client'' closes with it and the next call constructs a fresh one — no stale closed transport can be reused.

Full 4-test matrix all green (unit + live with real OpenRouter round trips, HERMES_LIVE_TESTS=1):

tests/run_agent/test_create_openai_client_kwargs_isolation.py      PASS
tests/run_agent/test_create_openai_client_reuse.py                 PASS (2)
tests/run_agent/test_sequential_chats_live.py                      PASS

Socket options verified on the live httpx transport:

_socket_options: [(1, 9, 1), (6, 4, 30), (6, 5, 10), (6, 6, 3)]
= (SO_KEEPALIVE=1, TCP_KEEPIDLE=30s, TCP_KEEPINTVL=10s, TCP_KEEPCNT=3)

Sequential-chat reproduction of the NousResearch#10933 failure was explicitly run against this patch — the defensive copy on main prevents the closed transport from leaking back into ''self._client_kwargs'', so every rebuild constructs a fresh transport.

Closes NousResearch#10324

What does this PR do?

Related Issue

Fixes #

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

How to Test

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform:

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

• This skill is broadly useful to most users (if bundled) — see Contributing Guide
• SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
• No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
• I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs