fix(credentials): isolate Codex 429 rotation and stale rechecks

[JIMVD2002]

Summary

Fix OpenAI Codex credential-pool recovery so a 429/401 is bound to the exact credential that sent the request instead of a potentially stale pool.current() entry.

This prevents a real usage_limit_reached response from one Codex account from persisting exhaustion metadata onto healthy sibling accounts.

What changed

• Thread the request credential as an exact api_key_hint through billing, rate-limit, and auth recovery paths.
• Fail closed when an authoritative hint does not match a pool entry.
• Refresh the exact request credential rather than a stale current pointer.
• Add an atomic Codex-only stale-cooldown probe state with bounded reclamation.
• Make both normal selection and delegated leases use the same disk-authoritative claim path.
• Persist only status metadata under the auth-store lock, preserving tokens, base URLs, priorities, and object identity.
• Use compare-and-set ownership for both late success and late failure so older requests cannot overwrite newer credential state.

Regression coverage

The tests cover:

• wrong-current-pointer 429 attribution;
• unmatched authoritative hints;
• stale in-memory pools versus newer on-disk state;
• selection and lease parity;
• concurrent probe claims;
• abandoned probe reclamation;
• late-success/newer-failure and late-failure/newer-success ordering;
• token/base-URL preservation;
• exact-token auth refresh;
• gateway and provider-boundary isolation.

Verification

Rebased/cherry-picked conflict-free onto current upstream main (2ea39daeb1). Patch ID matches the independently reviewed local deployment commit.

188 passed in 28.89s
py_compile: PASS
git diff --check: PASS

Live profile validation was performed separately with three real OpenAI Codex OAuth credentials using minimal streaming Responses calls. After label-bound token synchronization, all three returned HTTP 200 with valid stream events. No credentials or secret values are included in this PR.

[alt-glitch]

This was generated by AI during triage.

Related to open #43755 and #58738: all address request-credential attribution. This branch additionally fails closed on unknown keys and targets refresh by the exact key, but its remaining request-state capture needs maintainer review.

[teknium1]

Thanks for the thorough attribution coverage. The narrow premise is real on current main: agent/agent_runtime_helpers.py:798, :822, :846, and :946 omit the existing CredentialPool.api_key_hint mechanism (agent/credential_pool.py:1583-1602).

Problems

• agent/conversation_loop.py:1645 promotes a probe before examining the Responses status. agent/codex_runtime.py:918-927 returns status="failed" responses without raising, so a failed terminal response can transiently persist STATUS_OK before later processing handles it.
• agent/credential_pool.py:128-133 adds a one-hour Codex reprobe despite a provider-supplied reset. Current main honors last_error_reset_at (agent/credential_pool.py:354-362), and AGENTS.md:155-163 specifically cautions against cooldown reprobes of confirmed-empty buckets.

Suggested changes

• Keep the exact request-key attribution and exact-key refresh fixes, but separate/remove the forced cooldown probe behavior pending a maintainer decision.
• Promote a probe only after a completed response is accepted; add a response.failed regression.

Automated hermes-sweeper review.

[teknium1]

run_codex_stream() returns a final object for response.failed events (agent/codex_runtime.py:918-927), so this runs before that failed response is rejected downstream. Promote only after a completed response has been accepted, and add a response.failed regression so a failed probe cannot become ok.

[teknium1]

This deliberately bypasses the provider reset timestamp after one hour. Current main honors last_error_reset_at, and AGENTS.md:155-163 warns against re-probing confirmed-empty cooldown buckets. Please keep the attribution fix separate from this policy change unless a maintainer explicitly approves it.

[JIMVD2002]

Addressed the sweeper review in 17d3c776d1:

• removed the forced one-hour Codex cooldown reprobe and persisted probe/CAS machinery;
• removed the pre-status probe promotion path from conversation_loop.py;
• retained the narrow fix: recovery/exhaustion and OAuth refresh are bound to the exact credential that executed the failed request;
• unknown request keys now fail closed instead of mutating or refreshing pool.current();
• tightened test doubles so dynamic MagicMock attributes cannot falsely advertise exact-key support.

Validation after rebasing onto current main (e0240d7bf7):

104 passed in 14.02s
python -m py_compile: PASS
git diff --check origin/main...HEAD: PASS

The net PR diff is now limited to three files: agent/agent_runtime_helpers.py, agent/credential_pool.py, and tests/agent/test_credential_pool_routing.py. The cooldown-reprobe behavior is no longer part of this PR.

[DeadlySilent]

I think the remaining attribution boundary is slightly narrower than the current implementation suggests. recover_with_credential_pool() still reads agent.api_key while handling the failure. That field is shared mutable runtime state; with overlapping requests or a rotation between dispatch and error handling, it can identify a different credential from the one that actually sent the failed request.

A safer invariant is to capture the concrete credential at the transport dispatch boundary and carry it in attempt-local state through recovery. The attempt should close when the request completes, late worker callbacks should be ignored, and an observed empty or unmatched credential should fail closed rather than falling back to pool.current().

A useful regression would dispatch two overlapping requests with synthetic credentials A and B, rotate shared agent state before A fails, and assert that only A receives the failure metadata. The same test should cover a late callback after the attempt closes.

I can contribute that focused regression/attempt-local binding if it fits the intended scope of this PR.

[teknium1]

Thanks for the thorough work here — closing as redundant after verification against current main.

The core fixes this PR carries all landed via the recent attribution series with a kwarg-based design:

• Exact-key failure attribution at all four rotation sites → fix(agent): pass api_key_hint to mark_exhausted_and_rotate in credential pool recovery (#43755 salvage) #69553 (api_key_hint on mark_exhausted_and_rotate, agent/agent_runtime_helpers.py)
• Pre-exhausted 429 check resolving the entry that actually failed → fix(agent): pass api_key_hint to mark_exhausted_and_rotate in credential pool recovery (#43755 salvage) #69553
• Exact-key OAuth refresh on 401 → fix(credential-pool): attribute failures to the key that failed, not the shared current() pointer (#58738 salvage) #69843 (try_refresh_matching(api_key_hint=...))
• The unmatched-hint fall-through hazard → fix(credential-pool): stop lost-update cooldown erasure and unmatched-hint wrong-key quarantine (#65844 salvage) #70147 (rotate without marking when the hint matches no entry)

Your try_refresh_for_api_key/entry_for_api_key approach would now duplicate that parallel API, and the branch conflicts with the landed versions. The one debatable remnant — failing closed (aborting recovery) on an unknown key rather than rotating — trades away graceful recovery for OAuth token drift cases, so we kept the rotate-without-marking behavior instead.

Appreciate the investigation, @JIMVD2002 — it independently identified the same root causes the merged series fixed.