fix(auth): replace misleading "No Codex credentials" with rate-limit reset message

[bensnelldev]

Problem

When the OpenAI Codex (ChatGPT-subscription) OAuth credential hits its 5-hour usage cap, the Hermes gateway logs this repeatedly until the window resets:

WARNING gateway.run: Primary provider auth failed: No Codex credentials stored. Run `hermes auth` to authenticate. — trying fallback

Two things wrong with that message:

1. The credential IS stored. hermes auth list simultaneously and correctly shows openai-codex-oauth-1 oauth device_code rate-limited usage_limit_reached (429) (1h 12m left).
2. Re-authenticating wouldn't help. It's a rate-limit window; the credential is healthy and will resume working when the quota resets. Telling the user to run hermes auth is actively misleading.

The same misleading flow exists for xai-oauth, qwen-oauth, google-gemini-cli, nous, and anthropic pool entries — anywhere a single credential lives only in credential_pool.<provider> rather than the legacy top-level providers.<provider> location.

Root cause

In hermes_cli/runtime_provider.py resolve_runtime_provider():

1. pool.has_credentials() is True (one entry in credential_pool.openai-codex).
2. pool.select() returns None because the only entry is in exhaustion cooldown.
3. With no usable pool entry, control falls through to the provider-specific branch (if provider == "openai-codex": → resolve_codex_runtime_credentials()).
4. That helper calls _read_codex_tokens() which reads providers.openai-codex (the legacy storage location, which is empty for pool-only credentials).
5. Raises AuthError("No Codex credentials stored. Run \hermes auth` to authenticate.", code="codex_auth_missing", relogin_required=True)`.
6. Gateway catches it and logs the misleading line.

Fix

Detect the "pool has entries but none are usable" case before the legacy fall-through, and raise a structured AuthError carrying the soonest reset time. The gateway/CLI/cron-scheduler all use a new describe_primary_failure(error) helper to render the warning so it surfaces the actual reason.

Specifically:

• New module agent/credential_status.py — moves classify_exhausted_status and format_exhausted_status out of hermes_cli/auth_commands.py so non-CLI callers (resolver, gateway logger) can import them without pulling in CLI command code. Adds summarize_pool_exhaustion(pool) and format_pool_exhaustion_message(provider, summary). auth_commands.py re-exports the originals to preserve any external imports.
• AuthError gains optional kind, reset_at, retry_after fields plus a from_pool_exhaustion(provider, summary) classmethod that builds a message like openai-codex rate-limited (usage_limit_reached, 429) - resets in 1h 12m.
• Resolver inserts a pool-exhaustion check between the early pool.select() branch and the provider-specific legacy fall-throughs (hermes_cli/runtime_provider.py). Covers openai-codex, xai-oauth, qwen-oauth, google-gemini-cli, nous, anthropic.
• Gateway/CLI/cron-scheduler log via describe_primary_failure(auth_exc), producing e.g. primary provider openai-codex rate-limited (usage_limit_reached, 429) - resets in 1h 12m — trying fallback. Plain AuthError without kind keeps its existing log shape (back-compat).

Why not just teach _read_codex_tokens to read from the pool?

That helper is also called by refresh + persistence paths that expect the providers shape. Mixing reads creates a write/read asymmetry. The early pool.select() branch already handles healthy pool-only credentials correctly — the bug only ever surfaces in the cooldown fall-through path, which is exactly where the new check fires.

Affected providers

• openai-codex
• xai-oauth
• qwen-oauth
• google-gemini-cli
• nous
• anthropic

Before / after

Before, in gateway log:

WARNING gateway.run: Primary provider auth failed: No Codex credentials stored. Run `hermes auth` to authenticate. — trying fallback

After:

WARNING gateway.run: primary provider openai-codex rate-limited (usage_limit_reached, 429) - resets in 1h 12m — trying fallback

Fallback still triggers identically (e.g. through to Copilot via gh CLI). The behavior change is purely the message; downstream consumers that branch on AuthError itself are unaffected unless they opt-in to the new kind/reset_at fields.

Test plan

New tests:

• tests/agent/test_credential_status.py — 16 cases covering classify_exhausted_status, format_remaining, format_exhausted_status, summarize_pool_exhaustion (empty pool, any-healthy → None, all-rate-limited picking soonest reset, mixed prefers rate-limit label, all-auth-failed), and format_pool_exhaustion_message.
• tests/hermes_cli/test_runtime_provider_pool_exhaustion.py — resolver raises structured AuthError(kind="rate_limit", reset_at=…) when the codex pool is rate-limit-exhausted; raises kind="auth_failed" with relogin_required=True for auth-failure-only entries; no-op fall-through for non-pool providers (openrouter).
• tests/gateway/test_auth_fallback_log_message.py — asserts the new "rate-limited (resets in …)" wording appears in the gateway warning and "No Codex credentials stored" does not; plain AuthError still logs the generic shape.

Updated tests:

• tests/hermes_cli/test_auth_commands.py — monkeypatch retargeted from hermes_cli.auth_commands.time.time to agent.credential_status.time.time to match the formatter's new home.
• tests/gateway/test_session_model_override_routing.py — fixture upgraded from the legacy stand-in AuthError("No Codex credentials stored. …") to a realistic structured AuthError(kind="rate_limit", …). Asserted behavior (fallback triggers) is unchanged.

Suite run (touched-area):

245 passed in 17.55s  # gateway + hermes_cli + agent credential tests
367 passed in 20.59s  # cron (scheduler.py was touched)

🤖 Generated with Claude Code

[outsourc-e]

The auth fix itself looks good, but this PR is stacked on unrelated mobile dashboard/web work from #28127 (6fa1701bd). Please do not merge this branch as-is.

I opened a clean replacement PR with only the auth/runtime/logging slice plus the targeted regression tests:

• #28185

Local validation on the clean replacement:

• scripts/run_tests.sh tests/agent/test_credential_status.py tests/hermes_cli/test_runtime_provider_pool_exhaustion.py tests/gateway/test_auth_fallback_log_message.py tests/gateway/test_session_model_override_routing.py tests/hermes_cli/test_auth_commands.py
• 71 passed

[alt-glitch]

Superseded by #28185, which is a clean replacement of this PR without unrelated commits. Same auth pool-exhaustion fix. Related: #27448 (same problem area — exhausted pools falling through to singleton auth).

[DeadlySilent]

Hit this in a live Telegram gateway session today via /model switching to gpt-5.5 via OpenAI Codex.

Observed behavior:

• /model failed with: Could not resolve credentials for provider 'OpenAI Codex': No Codex credentials stored. Run hermes auth to authenticate.
• auth.json still had valid credential_pool.openai-codex entries with access + refresh tokens for multiple profiles.
• hermes auth list showed the openai-codex credentials, but with cached rate-limited usage_limit_reached / exhausted status.
• A live Codex usage probe showed quota had already reset and the relevant profiles were usable again.
• Running hermes auth reset openai-codex cleared the cached exhaustion state on 3 credentials.
• Immediately after that, /model could switch to Codex successfully.

So there seems to be one additional sharp edge beyond the misleading wording while currently rate-limited: cached exhausted status can continue to block /model after quota has reset, unless the user manually runs hermes auth reset openai-codex.

Suggested UX/behavior: when resolving openai-codex credentials and every stored pool entry is marked exhausted, Hermes should either live-probe/refresh expired cooldown state before declaring no usable credentials, or produce a message like “Codex credentials are present but marked rate-limited; run hermes auth reset openai-codex or wait until ” rather than saying no credentials are stored.

[teknium1]

Thanks for the detailed fix and tests. I would not merge this PR branch as-is.

Problems

• This PR has already been reviewed as stacked on unrelated mobile dashboard/web work; the existing review points to #28185 as the clean replacement. I verified #28185 is still open/unmerged with gh pr view 28185.
• Current main has moved around this area: f1422ff added Codex 429 classification via CODEX_RATE_LIMITED_CODE / is_rate_limited_auth_error (hermes_cli/auth.py:718, gateway/run.py:1416), and 69dfcdc added a Codex pool fallback in resolve_codex_runtime_credentials() (hermes_cli/auth.py:3749). Salvage needs to reconcile with those existing paths rather than applying this branch directly.
• Current main also has STATUS_DEAD pool entries (agent/credential_pool.py:63) that are excluded from availability (agent/credential_pool.py:1282). The proposed summarize_pool_exhaustion() only summarizes entries with last_status == STATUS_EXHAUSTED, so an all-DEAD pool would still bypass the structured message path.

Suggested changes

• Prefer the clean replacement PR (#28185) or a fresh narrow salvage branch.
• Fold the structured pool-status UX into the existing rate-limit/AuthError classification now on main.
• Include DEAD/all-unavailable pool cases in the tests if this is salvaged.

Automated hermes-sweeper review.

[teknium1]

Current main has STATUS_DEAD entries that are also unavailable to pool.select(); limiting the summary to STATUS_EXHAUSTED means an all-DEAD pool will still fall through to the legacy singleton-auth path. A salvage should account for DEAD explicitly or document why it is intentionally excluded.

[teknium1]

Current main already has CODEX_RATE_LIMITED_CODE/is_rate_limited_auth_error for quota failures. If this is salvaged, this new kind-based branch should be reconciled with that existing classifier rather than becoming a parallel rate-limit mechanism.

[teknium1]

Current main now logs CODEX_RATE_LIMITED_CODE separately in this handler. Salvage should preserve that behavior while adding pool-exhaustion wording, instead of replacing it wholesale with a new helper that does not know about the existing classifier.