Skip to content

fix(agent): avoid closing shared Anthropic client on interrupt - #51688

Closed
raymondyan-zhijie wants to merge 2 commits into
NousResearch:mainfrom
raymondyan-zhijie:fix/29507-anthropic-client-interrupt-fd-recycle
Closed

fix(agent): avoid closing shared Anthropic client on interrupt#51688
raymondyan-zhijie wants to merge 2 commits into
NousResearch:mainfrom
raymondyan-zhijie:fix/29507-anthropic-client-interrupt-fd-recycle

Conversation

@raymondyan-zhijie

Copy link
Copy Markdown
Contributor

Anthropic Messages uses a shared SDK client, unlike the request-local OpenAI-wire clients protected by the existing #29507 owner-thread close discipline. The stale/interrupt poll loop was still closing and rebuilding that shared client from a non-owner thread, which can release TLS file descriptors while a worker thread is still unwinding the SSL BIO. If SQLite reuses that descriptor, pending TLS bytes can clobber a database header.

Skip shared Anthropic client close/rebuild from non-streaming stale, non-streaming interrupt, and streaming interrupt paths. The in-flight SDK call is left to its timeout or stream context cleanup while the existing OpenAI request-local close path remains unchanged.

Add regression coverage for the Anthropic-compatible paths so future interrupt/stale changes do not reintroduce shared client close/rebuild.

What does this PR do?

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

How to Test

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform:

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

  • This skill is broadly useful to most users (if bundled) — see Contributing Guide
  • SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
  • No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
  • I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

Anthropic Messages uses a shared SDK client, unlike the request-local
OpenAI-wire clients protected by the existing NousResearch#29507 owner-thread close
discipline. The stale/interrupt poll loop was still closing and rebuilding
that shared client from a non-owner thread, which can release TLS file
descriptors while a worker thread is still unwinding the SSL BIO. If SQLite
reuses that descriptor, pending TLS bytes can clobber a database header.

Skip shared Anthropic client close/rebuild from non-streaming stale,
non-streaming interrupt, and streaming interrupt paths. The in-flight SDK call
is left to its timeout or stream context cleanup while the existing OpenAI
request-local close path remains unchanged.

Add regression coverage for the Anthropic-compatible paths so future
interrupt/stale changes do not reintroduce shared client close/rebuild.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/anthropic Anthropic native Messages API P2 Medium — degraded but workaround exists labels Jun 24, 2026
Anthropic Messages calls previously used the shared SDK client for the in-flight
request. Avoiding shared-client close from the poll thread fixed the FD recycle
risk, but left stale and interrupted Anthropic requests running until SDK
timeout.

Create per-request Anthropic clients for non-streaming and streaming calls,
using the same owner-thread close discipline as OpenAI-wire requests. Stranger
threads only abort sockets; the owning worker performs the SDK close in its
finally block. Streaming Anthropic interrupts now raise before
get_final_message() so cancelled streams do not drain to completion.

Extend the cascading-interrupt regression tests to assert that the shared client
is not closed, request-local clients are aborted/closed, and interrupted streams
do not call get_final_message().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the shared-client ownership issue. The premise is still present on current main: non-streaming stale/interrupt paths close agent._anthropic_client at agent/chat_completion_helpers.py:717-719 and :759-761, and streaming interrupt does so at :3109-3111.

Problems

  • Current main has three additional shared-Anthropic cleanup sites that this PR does not cover: mid-tool retry at agent/chat_completion_helpers.py:2834-2837, ordinary retry at :2894-2897, and stale-stream cleanup at :3080-3083. All call close() and _rebuild_anthropic_client() on the shared client.
  • The non-streaming dispatch was extracted after this PR's base (agent/chat_completion_helpers.py:449-457, commit 5c5dd6b7e), so this needs a current-structure transplant rather than a direct application.

Suggested changes

  • Carry the request-local Anthropic ownership discipline through all six current cleanup/interrupt paths and add regression cases for the three streaming retry/stale paths.

Automated hermes-sweeper review.

Comment thread run_agent.py
exc,
)

def _create_request_anthropic_client(self, *, reason: str) -> Any:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When salvaging this request-local client helper onto current main, route every current shared-Anthropic cleanup site through it. Current main still closes/rebuilds the shared client in streaming mid-tool retry, retry, and stale cleanup (agent/chat_completion_helpers.py:2836, 2896, 3082) in addition to the paths changed here.

@OutThisLife

Copy link
Copy Markdown
Collaborator

Carried this forward in #67238 for #67142 (the direct-Anthropic cron/executions.db corruption report). Your per-request-client approach was exactly right — I reimplemented it onto current main, which had drifted since this PR:

I also pushed the invariant a bit further — the shared _anthropic_client is now never closed from inside a request at all (including the worker retry-cleanup sites), since each attempt builds a fresh request-local client; that removes the last cross-request shared-client close. #28161 no-hang is preserved via the poll-thread socket abort.

Your commits are preserved via Co-authored-by (the originals were authored as "Claude Code" — credited to you as the human contributor). Will close as superseded once #67238 lands. Thanks @raymondyan-zhijie.

OutThisLife added a commit that referenced this pull request Jul 19, 2026
…-abort

fix(agent): request-local Anthropic clients so the stale/interrupt watchdog never corrupts SQLite (#67142, supersedes #51688)
@OutThisLife

Copy link
Copy Markdown
Collaborator

Landed via #67238 (merged) — your commits preserved via Co-authored-by. Closing as superseded. Thanks @raymondyan-zhijie.

randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…tchdog never corrupts SQLite (NousResearch#67142)

Direct-Anthropic requests used a single shared _anthropic_client, and the
stale/interrupt watchdog closed + rebuilt it from the poll (stranger) thread
at four sites (non-streaming stale/interrupt, streaming stale/interrupt).
Closing a client whose TLS socket a worker thread was still reading released
the FD from a stranger thread; the kernel recycled it under a live SSL BIO,
which then wrote a 24-byte TLS record into an unrelated SQLite header
(cron/executions.db), bricking every cron on the profile. Same shape as the
OpenAI-only NousResearch#29507 fix, but the Anthropic path never got the owner-thread
contract.

Extend the NousResearch#29507 ownership contract to Anthropic: build a per-request client
(_create_request_anthropic_client), register it with the request-client
holder tagged by kind, and route _close_request_client_once by kind — a
stranger thread only shuts the request client's sockets down
(_abort_request_anthropic_client), while the owning worker performs the SDK
close (_close_request_anthropic_client). The shared _anthropic_client is now
never closed from inside a request (streaming or non-streaming), including the
worker retry-cleanup sites, since each attempt builds a fresh request client.
The NousResearch#28161 no-hang guarantee is preserved: the poll-thread socket abort
unblocks the worker immediately.

Salvages the approach from NousResearch#51688 (@raymondyan-zhijie), reimplemented onto
current main (non-streaming dispatch was refactored into
_dispatch_nonstreaming_api_request; streaming grew _cancel_current_stream_attempt
and worker retry-cleanup sites). Tests updated to the request-local mechanism
(incl. replacing a banned source-reading test with a behavior test) plus new
regression coverage proving the watchdog aborts the request client and never
touches the shared client.

Co-authored-by: raymondyan-zhijie <32435458+raymondyan-zhijie@users.noreply.github.com>
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…owner-thread-abort

fix(agent): request-local Anthropic clients so the stale/interrupt watchdog never corrupts SQLite (NousResearch#67142, supersedes NousResearch#51688)
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
…tchdog never corrupts SQLite (NousResearch#67142)

Direct-Anthropic requests used a single shared _anthropic_client, and the
stale/interrupt watchdog closed + rebuilt it from the poll (stranger) thread
at four sites (non-streaming stale/interrupt, streaming stale/interrupt).
Closing a client whose TLS socket a worker thread was still reading released
the FD from a stranger thread; the kernel recycled it under a live SSL BIO,
which then wrote a 24-byte TLS record into an unrelated SQLite header
(cron/executions.db), bricking every cron on the profile. Same shape as the
OpenAI-only NousResearch#29507 fix, but the Anthropic path never got the owner-thread
contract.

Extend the NousResearch#29507 ownership contract to Anthropic: build a per-request client
(_create_request_anthropic_client), register it with the request-client
holder tagged by kind, and route _close_request_client_once by kind — a
stranger thread only shuts the request client's sockets down
(_abort_request_anthropic_client), while the owning worker performs the SDK
close (_close_request_anthropic_client). The shared _anthropic_client is now
never closed from inside a request (streaming or non-streaming), including the
worker retry-cleanup sites, since each attempt builds a fresh request client.
The NousResearch#28161 no-hang guarantee is preserved: the poll-thread socket abort
unblocks the worker immediately.

Salvages the approach from NousResearch#51688 (@raymondyan-zhijie), reimplemented onto
current main (non-streaming dispatch was refactored into
_dispatch_nonstreaming_api_request; streaming grew _cancel_current_stream_attempt
and worker retry-cleanup sites). Tests updated to the request-local mechanism
(incl. replacing a banned source-reading test with a behavior test) plus new
regression coverage proving the watchdog aborts the request client and never
touches the shared client.

Co-authored-by: raymondyan-zhijie <32435458+raymondyan-zhijie@users.noreply.github.com>
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
…owner-thread-abort

fix(agent): request-local Anthropic clients so the stale/interrupt watchdog never corrupts SQLite (NousResearch#67142, supersedes NousResearch#51688)
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…tchdog never corrupts SQLite (NousResearch#67142)

Direct-Anthropic requests used a single shared _anthropic_client, and the
stale/interrupt watchdog closed + rebuilt it from the poll (stranger) thread
at four sites (non-streaming stale/interrupt, streaming stale/interrupt).
Closing a client whose TLS socket a worker thread was still reading released
the FD from a stranger thread; the kernel recycled it under a live SSL BIO,
which then wrote a 24-byte TLS record into an unrelated SQLite header
(cron/executions.db), bricking every cron on the profile. Same shape as the
OpenAI-only NousResearch#29507 fix, but the Anthropic path never got the owner-thread
contract.

Extend the NousResearch#29507 ownership contract to Anthropic: build a per-request client
(_create_request_anthropic_client), register it with the request-client
holder tagged by kind, and route _close_request_client_once by kind — a
stranger thread only shuts the request client's sockets down
(_abort_request_anthropic_client), while the owning worker performs the SDK
close (_close_request_anthropic_client). The shared _anthropic_client is now
never closed from inside a request (streaming or non-streaming), including the
worker retry-cleanup sites, since each attempt builds a fresh request client.
The NousResearch#28161 no-hang guarantee is preserved: the poll-thread socket abort
unblocks the worker immediately.

Salvages the approach from NousResearch#51688 (@raymondyan-zhijie), reimplemented onto
current main (non-streaming dispatch was refactored into
_dispatch_nonstreaming_api_request; streaming grew _cancel_current_stream_attempt
and worker retry-cleanup sites). Tests updated to the request-local mechanism
(incl. replacing a banned source-reading test with a behavior test) plus new
regression coverage proving the watchdog aborts the request client and never
touches the shared client.

Co-authored-by: raymondyan-zhijie <32435458+raymondyan-zhijie@users.noreply.github.com>
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…owner-thread-abort

fix(agent): request-local Anthropic clients so the stale/interrupt watchdog never corrupts SQLite (NousResearch#67142, supersedes NousResearch#51688)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/anthropic Anthropic native Messages API sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants