fix(agent): request-local Anthropic clients so the stale/interrupt watchdog never corrupts SQLite (#67142, supersedes #51688)

[OutThisLife]

Summary

Salvages #51688 (@raymondyan-zhijie) for #67142 — the direct-Anthropic gateway corruption where a stale/interrupt watchdog closed the shared Anthropic client from the poll thread and recycled a live TLS FD into a SQLite header, bricking cron/executions.db (every due cron then failed at create_execution()).

Root cause: direct-Anthropic requests share one _anthropic_client, and four watchdog sites closed + rebuilt it from the poll (stranger) thread:

• non-streaming stale (interruptible_api_call)
• non-streaming interrupt
• streaming stale (interruptible_streaming_api_call)
• streaming interrupt

Closing a client whose TLS socket a worker thread was still reading releases the FD from a stranger thread; the kernel recycles it under a live SSL BIO, which writes a 24-byte TLS record over an unrelated SQLite header. This is the same shape as #29507, but the OpenAI-only owner-thread fix never covered the Anthropic path.

Fix — extend the #29507 ownership contract to Anthropic:

• _create_request_anthropic_client / _close_request_anthropic_client / _abort_request_anthropic_client on the agent (mirror the OpenAI request-client helpers; direct + Bedrock + 1M-beta-drop construction).
• The request-client holder is tagged by kind (openai / anthropic_messages); _close_request_client_once routes by kind — a stranger thread only shutdown()s the request client's sockets, the owning worker performs the SDK close.
• _anthropic_messages_create(..., client=...) and _call_anthropic(request_client) use the per-request client instead of the shared one.
• The shared _anthropic_client is never closed from inside a request — including the worker retry-cleanup sites, since each attempt builds a fresh request client. The two worker-owned retry cleanups the issue flagged stay worker-owned (they now close the request client).

#28161 no-hang preserved: the poll-thread socket abort unblocks the worker immediately; the next attempt builds a fresh client, so no dead shared pool lingers.

Why supersede instead of merging #51688

#51688 had the right approach but predated two refactors on main:

• Non-streaming dispatch was extracted into _dispatch_nonstreaming_api_request(make_client=...); the per-request Anthropic client is now threaded through a kind-aware make_client there rather than an inline if/elif.
• Streaming grew _cancel_current_stream_attempt and worker retry-cleanup sites (Anthropic streaming: stale/retry paths call _replace_primary_openai_client, causing 15-min hang on stuck streams #28161 lineage) that fix(agent): avoid closing shared Anthropic client on interrupt #51688 never saw.

Reimplemented onto current main and reconciled with those; authorship preserved via Co-authored-by. (The original commits were authored as "Claude Code" — credited to the human contributor.)

Novel changes in the mix

• The coherent invariant "the shared _anthropic_client is never closed from within a request" — fix(agent): avoid closing shared Anthropic client on interrupt #51688 left the worker retry-cleanup sites closing/rebuilding the shared client; with request-local clients that's redundant churn and still a cross-request close, so it's removed. All request cleanup goes through the request-local client.
• Replaced a banned source-reading test (test_interruptible_rebuilds_anthropic_client, which asserted the now-removed rebuild-on-interrupt behavior via inspect.getsource) with a real behavior test.
• New regression tests proving both Anthropic stale-stream watchdog can still corrupt SQLite via TLS FD reuse #67142 properties for non-streaming stale + interrupt (watchdog aborts the request client, shared client untouched, worker unblocks and closes its own client). Updated test_28161 / test_streaming / circuit-breaker tests to the request-local mechanism.

Test plan

• scripts/run_tests.sh tests/run_agent/ tests/agent/ — 8174 pass, 3 fail (pre-existing TestRunOauthSetupToken failures, identical on clean main, unrelated).
• test_28161_anthropic_stream_pool_cleanup.py, test_stream_stale_circuit_breaker.py, test_streaming.py, test_cascading_interrupt_6600.py, test_run_agent.py — all green.
• grep confirms no _anthropic_client.close() remains in any request path in chat_completion_helpers.py.
• Manual: direct-Anthropic cron with a mcp_servers.<srv>.timeout-style long call; force a stale/interrupt and confirm the ledger stays integrity-clean and the worker unblocks promptly.

Fixes #67142

[alt-glitch]

This was generated by AI during triage.

Related to #67210 and #51688: this salvage uses request-local Anthropic clients across the watchdog paths, while #67210 preserves the shared-client model with owner-thread teardown. Both address #67142 but differ in contract and scope; maintainer selection is needed.