fix(auth): harden Codex credential pool recovery

[Qwinty]

What does this PR do?

Fixes a cluster of Codex credential-pool failure modes where Hermes can keep reporting openai-codex as exhausted even after a usable pooled credential exists.

The root cause is that pool exhaustion state is persistent, but runtime recovery had a few gaps:

• an exhausted pool could fall through to singleton Codex auth and produce a misleading/auth-stale runtime path;
• Codex usage_limit_reached status could stay persisted incorrectly when no concrete reset window exists, while concrete future reset timestamps from failed model requests must remain authoritative;
• fresh Responses 429 reset timestamps must remain authoritative until they elapse, even when the auxiliary live-usage probe reports availability;
• invalidated Codex tokens should not stay in rotation;
• chatgpt_account_id can identify a workspace/account, not the user-level usage bucket, so workspace members need to be grouped by (chatgpt_account_id, JWT sub) for quota propagation.

This is intentionally a single draft PR because the final workspace-member quota fix depends on the live-usage reconciliation path introduced earlier in this stack.

Related Issue

Related to #27448 and #30802. No separate issue yet; this was reproduced from live openai-codex pool state and covered with regression tests.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• hermes_cli/runtime_provider.py: report exhausted pools explicitly instead of silently falling through to singleton OAuth, and include Codex duplicate quota identity diagnostics.
• hermes_cli/auth_commands.py: surface DEAD credentials in hermes auth list instead of making them look selectable.
• agent/credential_pool.py: reconcile exhausted Codex entries against live usage, keep concrete 429 reset windows authoritative until their reset timestamps elapse, quarantine invalidated Codex tokens, and group shared workspace entries by quota identity.
• Runtime entry points (agent/agent_init.py, run_agent.py, gateway/run.py, cron/TUI/CLI helpers): preserve credential_pool_entry_id so 429 recovery marks the exact credential used by the failed request.
• Tests cover stale usage reconciliation, fresh cooldown holds, duplicate quota propagation, workspace-member separation, invalid token quarantine, and runtime exhausted-pool errors.

How to Test

1. /usr/local/lib/hermes-agent/venv/bin/python -m pytest -q -o addopts='' tests/agent/test_credential_pool.py tests/hermes_cli/test_runtime_provider_resolution.py tests/hermes_cli/test_auth_commands.py tests/run_agent/test_run_agent.py::TestCredentialPoolRecovery
2. /usr/local/lib/hermes-agent/venv/bin/python -m ruff check agent/credential_pool.py hermes_cli/runtime_provider.py tests/agent/test_credential_pool.py tests/hermes_cli/test_runtime_provider_resolution.py tests/hermes_cli/test_auth_commands.py tests/run_agent/test_run_agent.py agent/agent_init.py agent/agent_runtime_helpers.py gateway/run.py
3. Live reproduction proof from production: before the quota-identity fix, Hermes grouped atobarrientos@gmail.com and aashish44khanal@gmail.com (2) together because they shared chatgpt_account_id=1538301a-...; live usage showed different user_id/usage windows, and after this change hermes auth list openai-codex selected the usable entry instead of reporting all 8 pool entries unavailable.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this is not a duplicate
• My PR contains only changes related to this fix/feature
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes
• I've tested on Ubuntu 24.04

Documentation & Housekeeping

• I've updated relevant documentation — N/A
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact — runtime/auth logic only, path-neutral
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Focused local validation:

226 passed, 2 warnings in 16.83s
All checks passed!

[Qwinty]

CI follow-up: the first run exposed a real missing helper in gateway/run.py (_load_credential_pool_for_provider). I restored the helper that session runtime overrides need and pushed 839820443.

Local verification after the fix:

• pytest -q -o addopts= tests/gateway/test_session_model_override_routing.py tests/agent/test_credential_pool.py tests/hermes_cli/test_runtime_provider_resolution.py tests/run_agent/test_run_agent.py::TestCredentialPoolRecovery - 209 passed
• ruff check ... on the changed auth/gateway files/tests - passed
• git diff --check upstream/main...HEAD - passed

[Qwinty]

Nix Ubuntu failed on a transient GitHub API 401 while fetching a public flake input (jeslie0/npm-lockfile-fix). I do not have permission to rerun upstream Actions directly, so I amended the last commit without changing the diff and force-pushed 03f47be49 to retrigger CI.

[jsboige]

Reviewed this PR focusing on security, error handling, race conditions, and test coverage.

Overall: well-structured fix. The credential_id threading through the recovery pipeline is clean. No secrets leak into logs (entries identified by label/id prefix only). JWT claim decoding for quota identity is purely local -- no network exposure.

One WARNING: _fetch_codex_entry_usage_status (network call, up to 8s timeout) runs inside the pool lock in _reconcile_codex_usage_unlocked, which is called from _select_unlocked. Under contention, a slow Codex usage endpoint blocks all pool operations for the lock duration. Consider moving the usage probe outside the lock and reconciling after, or capping the timeout lower for the hot path.

Positive notes: (1) pool_unavailable flag correctly prevents silent fallthrough to singleton auth -- the core bug fix is sound. (2) Grace period logic for fresh 429 reset timestamps prevents tight retry loops -- well designed. (3) Workspace-member quota separation via (chatgpt_account_id, JWT sub) is the right granularity. (4) 648 lines of new tests covering stale reconciliation, fresh cooldown, duplicate propagation, independent subjects, token quarantine, and runtime exhausted-pool errors -- thorough.

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

[Qwinty]

CI is green after the no-diff retrigger: tests, ruff/ty, Nix on Ubuntu/macOS, supply-chain, history/attribution, docs/docker jobs all pass on 03f47be49.

[Qwinty]

Rebased on current upstream/main and addressed the review warning about the Codex live-usage probe running in the pool-selection path.

Changes after rebase:

• Preserved both upstream api_key_hint and this PR's credential_id targeting in mark_exhausted_and_rotate, preferring credential_id when available.
• Capped the opportunistic Codex usage probe timeout to 2s even when HERMES_CODEX_USAGE_TIMEOUT_SECONDS is set higher, so a slow usage endpoint cannot hold the pool lock for the previous 8s default.
• Added regression coverage for the timeout cap.
• Fixed the pre-exhausted rate-limit rotation path to use the same credential-id-aware helper.

Local verification:

• python -m pytest -q -o addopts='' tests/gateway/test_session_model_override_routing.py tests/agent/test_credential_pool.py tests/hermes_cli/test_runtime_provider_resolution.py tests/run_agent/test_run_agent.py::TestCredentialPoolRecovery - 229 passed
• python -m ruff check agent/credential_pool.py agent/agent_runtime_helpers.py tests/agent/test_credential_pool.py tests/run_agent/test_run_agent.py - passed
• git diff --check upstream/main...HEAD - passed

CI has been retriggered on 451f388e0.

[Qwinty]

Follow-up from the live Codex pool reproduction on vps_new: Responses/Codex returned a concrete future reset window for usage_limit_reached while the live wham/usage probe reported the same entry as available. The previous bounded-grace behavior let the probe clear that entry after 5 minutes, which can reselect the same account and hit the same model-endpoint 429 again.

Pushed fb4e4f00e fix(auth): trust Codex usage-limit reset windows:

• keep a concrete future last_error_reset_at from a Codex usage-limit failure authoritative until it elapses;
• still allow live usage reconciliation for stale/no-reset exhaustion and elapsed reset windows;
• remove the now-misleading HERMES_CODEX_USAGE_LIMIT_RECONCILE_GRACE_SECONDS path;
• add regression coverage for the conflicting Responses 429 future reset vs usage probe available case.

Validation:

• /usr/local/lib/hermes-agent/venv/bin/python -m pytest -q -o addopts='' tests/agent/test_credential_pool.py tests/hermes_cli/test_runtime_provider_resolution.py tests/run_agent/test_run_agent.py::TestCredentialPoolRecovery - 226 passed, 2 warnings
• /usr/local/lib/hermes-agent/venv/bin/python -m ruff check agent/credential_pool.py tests/agent/test_credential_pool.py agent/agent_runtime_helpers.py tests/run_agent/test_run_agent.py hermes_cli/runtime_provider.py tests/hermes_cli/test_runtime_provider_resolution.py gateway/run.py - passed
• git diff --check - passed

[Qwinty]

Follow-up from live production repro on 2026-05-26:

• codex-auth and Hermes' own wham/usage probe showed some Codex entries as allowed=true, while the pool still held stale usage_limit_reached cooldowns from earlier Responses 429s.
• Added bca95db1b fix(auth): clear Codex cooldown when usage allows requests so future reset windows remain authoritative for heuristic usage availability, but an explicit allowed=true response from Codex usage can clear stale cooldown state.
• Production validation: direct gpt-5.5 high streaming probes completed for aashish44khanal@gmail.com and atobarrientos@gmail.com; targeted tests pass (84 passed in the PR worktree, Codex subset 6 passed in production).

Production gateway has the same fix cherry-picked as 14c149340 and the stale pool entries were reconciled; #2 and #5 are back in rotation.

[teknium1]

Thanks for the thorough recovery coverage. The central premise is still present on current main: hermes_cli/runtime_provider.py:1698-1705 leaves an exhausted pool with no selected entry, then :1784-1795 falls through to singleton Codex auth. The live-usage reconciliation and exact-entry recovery work therefore remain valuable.

Problems

• agent/credential_pool.py:379 adds HERMES_CODEX_USAGE_TIMEOUT_SECONDS. The project policy at AGENTS.md:102-106 reserves .env / HERMES_* for secrets; behavioral timeouts belong in config.yaml. Given the code caps this probe at two seconds, a fixed internal constant appears sufficient.
• tests/agent/test_credential_pool.py:18 and :38 both define _jwt_with_claims; the latter silently overwrites the former.

Suggested changes

• Remove the new environment-variable configuration path, or wire a documented config setting if it must be user-configurable.
• Deduplicate the test helper before salvage.

Automated hermes-sweeper review.

[Qwinty]

Queue hygiene note after portfolio refresh (2026-07-19):

This PR is still valuable, but it is the broadest dirty item in my open set (+1300 across auth/runtime/CLI surfaces) and currently conflicts with current main in agent/credential_pool.py and agent/agent_runtime_helpers.py.

I refreshed the narrower dirty PRs first. I’ll do a dedicated rebase/salvage pass for this one next rather than force a risky mega-resolve in the same batch.

[Qwinty]

Dedicated current-main salvage/rebase completed in ea3028803.

The three conflicts were resolved by preserving current main behavior while retaining this PR's exact-entry recovery:

• kept main's api_key_hint targeting and no-available-entry log throttling;
• added credential_pool_entry_id targeting across runtime entry points;
• preserved try_refresh_matching() and combined ID/key fallback for stale or reloaded pool instances;
• retained current pool selection refresh controls while adding Codex live-usage reconciliation.

Independent verification after conflict resolution:

• focused auth/pool/runtime suite plus routing, gateway override, fallback isolation, and background-review regressions: 388 passed
• earlier full PR-focused suite: 320 passed
• ruff, Python compilation, and git diff --check: passed
• all prior review threads remain resolved.

[Qwinty]

Late findings follow-up

Pushed 3ca30f89b for Codex pool recovery hardening:

• when select() runs under a live asyncio loop, Codex live-usage reconciliation is offloaded to a worker instead of blocking the event loop
• clear stale _credential_pool_entry_id on provider rebinds (init mismatch, model switch, fallback attach, primary restore)
• added regressions:
  • tests/agent/test_codex_pool_selection_no_network.py
  • tests/run_agent/test_credential_pool_entry_id_rebind.py

Local verification: credential-pool + switch/fallback/restore suites green.

[Qwinty]

Final independent-review follow-up

Pushed 64b0ead9a to fix the false-exhaustion blocker found in independent review:

• CredentialPool.select() keeps its synchronous contract and returns a credential after successful Codex live-usage reconciliation
• every async gateway/slash runtime-resolution call site now runs off the event loop via asyncio.to_thread
• provider switches and fallback pool attaches bind the exact entry ID that supplied the resolved runtime key, without a second selection/probe

Local verification: 189 passed; ruff clean.

[potalora]

Fresh production reproduction of the stale Codex exhaustion path on Hermes v0.19.0 (2026.7.20), upstream f1345290, Docker, Python 3.13.5.

Observed

• openai-codex / gpt-5.6-sol was the primary provider/model.
• One pooled Codex OAuth credential had been persisted as:

last_status: exhausted
last_status_at: 2026-07-27 09:10:20 UTC
last_error_code: 429
last_error_reason: usage_limit_reached
last_error_message: The usage limit has been reached
last_error_reset_at: 2026-08-03 05:17:04 UTC

• On July 29, the ChatGPT Codex usage dashboard showed 96% weekly quota remaining.
• Nevertheless, every gateway turn failed during primary credential resolution without attempting Codex:

Primary provider rate-limited (429): Codex provider quota exhausted (429); retry after ... Credentials are still valid. — trying fallback
Fallback provider resolved: anthropic model=claude-sonnet-4-6

• This silently routed interactive Telegram traffic to a metered Anthropic fallback.

Recovery and verification

I ran the supported reset:

hermes auth reset openai-codex
# Reset status on 1 openai-codex credentials

I then copied the config/auth store into an isolated temporary HERMES_HOME, removed all fallback providers there, and made a Codex-only one-shot request:

hermes chat -q 'Reply with exactly CODEX_OK and nothing else.' \
  --provider openai-codex -m gpt-5.6-sol

It returned:

CODEX_OK

No reauthentication or token replacement was required. Clearing only the persisted exhaustion fields restored the same OAuth credential immediately.

Why this is useful evidence for this PR

This independently reproduces the false-positive exhaustion case described in the PR: a concrete future last_error_reset_at remained authoritative locally even though the same credential was currently usable. Runtime selection did not perform a live availability reconciliation before falling back.

The manual workaround is effective but non-obvious:

hermes auth reset openai-codex

This reproduction supports the PR's live-usage reconciliation work and its requirement that an explicit usable/allowed result clear stale persisted cooldown state. It also demonstrates direct billing impact when the configured fallback is a metered provider.