fix(auxiliary): evict Codex auxiliary client on timeout

[wesleyhere]

What does this PR do?

Fixes a race in the Codex auxiliary Responses timeout cleanup path that can leave a poisoned client in the auxiliary cache.

Root cause

_CodexCompletionsAdapter.create() in agent/auxiliary_client.py has two paths that detect the same total-timeout deadline:

1. Timer thread — threading.Timer(total_timeout, _close_client_on_timeout) fires from a separate thread when the deadline elapses. _close_client_on_timeout() sets timed_out, closes the underlying OpenAI client, and calls _evict_cached_client_instance() to drop the cached wrapper (added by fix(auxiliary): evict cached client on timeout/connection error #23482; extended to async wrappers by fix(auxiliary): evict async wrappers on poisoned client (follow-up to #23482) #23931).

2. In-loop deadline check — _check_cancelled() is invoked between stream events inside for _event in stream:. When time.monotonic() >= deadline, it previously set timed_out only, then raised TimeoutError directly — skipping the close and the cache eviction.

The two paths are armed at the same threshold but run on different threads, so they race. On tight timeouts the in-loop check wins deterministically: stream iteration drives _check_cancelled() on every event, while the timer thread may not be scheduled in time. When path 2 wins, the timeout exits without running the cleanup path that closes the Codex client and evicts cached wrappers. That leaves cache state dependent on a thread scheduling race: later auxiliary calls can reuse a stale or poisoned wrapper instead of rebuilding cleanly, surfacing as openai.APIConnectionError: Connection error (the cascade originally tracked in #23432).

Fix

Route the in-loop deadline check through _close_client_on_timeout() so both detection paths perform the same cleanup (close + cache eviction) before raising TimeoutError. _close_client_on_timeout() is already idempotent — timed_out.set() is set-or-no-op, client.close() is safe to call twice, and _evict_cached_client_instance() no-ops on an already-evicted entry — so the timer firing afterward causes no harm.

 def _check_cancelled() -> None:
     if deadline is not None and time.monotonic() >= deadline:
-        timed_out.set()
+        _close_client_on_timeout()
         raise TimeoutError(_timeout_message())

Related Issue

Refs #23617, #21761, #22986. No standalone issue; the _check_cancelled race is covered by the regression test added here and by the related work below from broader angles.

Prior art / related work:

• Refs fix(agent): avoid Codex stream for compression summaries #23617. That PR includes the same _check_cancelled() cleanup change, but bundles it with a larger threaded stream consumer, Codex compression routing changes, and task-aware auto cache keys. This PR is the minimal isolated fix for the timeout cleanup race. If fix(agent): avoid Codex stream for compression summaries #23617 lands first, this PR can be closed; if maintainers prefer the smaller fix, it can land independently and fix(agent): avoid Codex stream for compression summaries #23617 can rebase/drop this line.
• Refs fix(auxiliary): recover Codex stream drops #21761. That PR handles transient Codex auxiliary stream drops and fallback streaming recovery.
• Refs [Bug] Codex APIConnectionError retry rate ~8x higher post-v0.13.0; persists with #12953 applied; suspected commit 5533ad764 strict stream-timeout enforcement #22986. That issue discusses increased Codex APIConnectionError rates around stream-timeout enforcement; this PR addresses one narrow cleanup race in that area.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/auxiliary_client.py: call _close_client_on_timeout() when _check_cancelled() detects the Codex auxiliary total-timeout deadline, so timeout cleanup consistently closes and evicts the cached client wrapper.
• tests/agent/test_auxiliary_client.py: add deterministic regression coverage for the loop-detected timeout path by disabling the timer callback and forcing _check_cancelled() to win.
• No behavior change on the happy path: _close_client_on_timeout() only runs when the deadline is exceeded, and it is already used by the timer path. Calling it from _check_cancelled() makes both timeout detection paths perform the same cleanup.

How to Test

1. Run the focused regression coverage:

   scripts/run_tests.sh tests/agent/test_auxiliary_client.py -k 'CodexAuxiliaryAdapterTimeout or AuxiliaryClientPoisonedCacheEviction'

2. Expected result:

   ▶ running pytest with 4 workers, hermetic env, in /Users/wesley/dev/oss/hermes-agent
     (TZ=UTC LANG=C.UTF-8 PYTHONHASHSEED=0; all credential env vars unset)
   ============================= test session starts ==============================
   platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
   rootdir: /Users/wesley/dev/oss/hermes-agent
   configfile: pyproject.toml
   plugins: anyio-4.12.1, xdist-3.8.0, split-0.11.0, asyncio-1.3.0
   asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
   created: 4/4 workers
   4 workers [10 items]
   
   ..........                                                               [100%]
   ============================== 10 passed in 1.19s ==============================
   

3. Optional wider check before requesting review:

   scripts/run_tests.sh tests/agent/test_auxiliary_client.py

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run the focused scripts/run_tests.sh target and all selected tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS, Python 3.12.12

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) - N/A
• I've updated cli-config.yaml.example if I added/changed config keys - N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows - N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide - N/A; this changes timeout cleanup logic only
• I've updated tool descriptions/schemas if I changed tool behavior - N/A

For New Skills

N/A

[teknium1]

This has already landed on main in the same cleanup path. Automated hermes-sweeper review found the Codex auxiliary in-loop timeout now routes through the timeout closer before raising.

Evidence:

• agent/auxiliary_client.py:801 now has _check_cancelled() call _close_client_on_timeout() when the deadline is exceeded, instead of only setting timed_out.
• agent/auxiliary_client.py:782 shows _close_client_on_timeout() closes the underlying client and calls _evict_cached_client_instance(self._client), matching this PR's requested close + eviction behavior.
• git blame -L 782,805 -- agent/auxiliary_client.py points the key _check_cancelled() line change to 89a3d038cfb289ce73b9d7aac9b0b7ca85a018f0, which is contained in release tag v2026.5.28.
• Current tests include tests/agent/test_auxiliary_client.py:3276, which asserts a Codex timeout closes the inner client and evicts the cached wrapper.

Thanks for isolating the race; the fix is now covered by current main.