fix(anthropic): abort watchdog TLS sockets from a stranger thread, close from the worker (#67142)

[HexLab98]

Summary

Fixes #67142 — a direct-Anthropic cron request reproduced the same 24-byte TLS/SQLite FD-reuse corruption shape as #29507, on api_mode == "anthropic_messages".

The outer stale/interrupt watchdog runs on the poll thread — a stranger thread relative to the worker driving _anthropic_client.messages.stream(...). On origin/main all four outer Anthropic watchdog branches called _anthropic_client.close() from that stranger thread:

• agent/chat_completion_helpers.py non-streaming stale watchdog + interrupt
• agent/chat_completion_helpers.py streaming stale watchdog + interrupt

Closing the client there races the worker's still-live SSL BIO. When the kernel recycles the just-freed TLS socket FD into an unrelated open() (e.g. cron/executions.db), a pending 24-byte TLS application-data record is flushed into that file, clobbering SQLite header bytes 5..28 and yielding sqlite3.DatabaseError: file is not a database. Because create_execution() runs before executor dispatch, every subsequent due cron on that profile then fails at the tick boundary.

This extends the #29507 owner-thread contract to the shared Anthropic client:

• Adds _abort_anthropic_client() (companion to _abort_request_openai_client) — a stranger-thread abort that only shutdown(SHUT_RDWR)s the client's httpx pool sockets and never releases the FD.
• Both the non-streaming and streaming loops stamp the worker thread id and route every Anthropic teardown through an owner-aware helper: a stranger-thread call aborts sockets (unblocking the worker) and defers the close+rebuild; the owning worker performs close() + _rebuild_anthropic_client() from its own thread (in its retry cleanup, or a finally drain on interrupt), where FD release is safe.

Both properties requested in the issue hold:

1. The outer stale/interrupt watchdog never releases the active Anthropic TLS socket FD from a stranger thread.
2. The worker still unblocks promptly and performs the close/rebuild from its own thread — preserving the Anthropic streaming: stale/retry paths call _replace_primary_openai_client, causing 15-min hang on stuck streams #28161 no-hang behavior (the abort unblocks a stuck stream, and the dead pool is still rebuilt).

The two in-worker retry-cleanup closes remain worker-owned.

Test plan

• scripts/run_tests.sh tests/run_agent/test_67142_anthropic_watchdog_fd_ownership.py — new suite proving the ownership contract (socket-level abort, streaming stale + interrupt, non-streaming stale).
• scripts/run_tests.sh tests/run_agent/test_28161_anthropic_stream_pool_cleanup.py tests/run_agent/test_stream_stale_circuit_breaker.py tests/run_agent/test_streaming.py — updated regressions still assert no OpenAI-primary replace and no 15-minute hang, now via the abort path.
• Broader regression: test_tls_fd_recycle_corruption, test_cascading_interrupt_6600, test_stream_interrupt_retry, test_non_stream_stale_timeout, test_bedrock_interrupt_post_worker, test_cron_inline_api_call_62151, test_stream_single_writer_65991, test_partial_stream_finish_reason, test_stream_stale_breaker_reset, test_moa_streaming, test_create_openai_client_reuse — all green.

Infographic

[alt-glitch]

This was generated by AI during triage.

Related to #67142, merged #30858, and open #51688. This extends the owner-thread FD-release discipline to the shared Anthropic client; it is not a duplicate of the prior request-client implementation.

[teknium1]

Thanks for extending the #29507 ownership discipline to the shared Anthropic path. The premise is confirmed on current main: the outer streaming stale and interrupt watchdogs still call the shared-client close/rebuild path at agent/chat_completion_helpers.py:3591-3595 and 3625-3629 (remote main 614dc194).

Problems

• The deferred finalizer closes agent._anthropic_client by current attribute lookup (agent/chat_completion_helpers.py:609 and 2519 in PR head), not the instance aborted by the watchdog. The stale non-streaming path returns after only t.join(timeout=2.0) (agent/chat_completion_helpers.py:958-972), so a late worker can close/rebuild a client that a subsequent request has begun using. Bind teardown to the call-owned client, or prevent a new shared-client request until the owning worker has drained.
• The added suite has non-streaming stale coverage but no non-streaming interrupt case (tests/run_agent/test_67142_anthropic_watchdog_fd_ownership.py:267-312), although the PR changes that branch too.

Suggested changes

• Add a delayed-worker regression that demonstrates a late finalizer cannot touch a later request's client.
• Add the missing non-streaming interrupt ownership regression.

Automated hermes-sweeper review.

[teknium1]

This must retain the specific client that the watchdog aborted. The stale path returns after a two-second join, so a late worker finalizer can otherwise close whatever agent._anthropic_client a later request has installed or is using.

[teknium1]

Please add the non-streaming interrupt case as well: this suite covers non-streaming stale handling, but the PR also changes the non-streaming interrupt watchdog branch.

[OutThisLife]

Closing as superseded by #67238 (merged), which already fixes #67142 (same class: Anthropic stale/interrupt watchdog closing the shared client from a stranger thread → TLS FD recycled into SQLite).

#67238 takes the request-local client approach (salvaging #51688) rather than abort-from-stranger-thread / close-from-worker. Please reopen only if you've reproduced a remaining corruption path on current main after that merge.