fix(gateway): consolidate Codex quota error handling

[konsisumer]

What does this PR do?

Consolidates the competing fixes for #32790 so Codex OAuth quota exhaustion is no longer surfaced as missing credentials. The gateway now uses AuthError.relogin_required to avoid logging transient upstream failures as authentication failures, and pool-only openai-codex credentials that are all in active 429 cooldown now raise the existing Codex rate-limit error instead of codex_auth_missing.

Supersedes

Supersedes #32881, #34325

Related Issue

Fixes #32790

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• hermes_cli/auth.py: detects the pool-only case where every token-bearing openai-codex credential is in a live 429 cooldown and raises CODEX_RATE_LIMITED_CODE with retry guidance instead of reusing the missing-credentials error.
• gateway/run.py: logs non-relogin AuthErrors as transient provider unavailability while preserving auth-failed wording for real credential failures.
• tests/hermes_cli/test_auth_codex_provider.py: covers all-429 pool cooldowns, 401 cooldowns, and mixed 429/401 pools.
• tests/gateway/test_auth_fallback.py: covers transient-vs-auth gateway log wording and config-provider fallback labels.

How to Test

1. HERMES_HOME=/private/tmp/pr-consolidate-1d689186-hermes-home /opt/homebrew/bin/timeout -k 30 480 pytest tests/gateway/test_auth_fallback.py tests/hermes_cli/test_auth_codex_provider.py tests/hermes_cli/test_auth_codex_self_heal.py tests/hermes_cli/test_runtime_provider_resolution.py tests/agent/test_credential_pool_routing.py tests/cron/test_codex_execution_paths.py tests/run_agent/test_run_agent_codex_responses.py tests/test_account_usage.py -q --timeout=60 (266 passed)
2. /opt/homebrew/bin/timeout -k 30 480 sh -c 'pytest tests/ -q -x --timeout=60 "$@"' sh aborted during collection because local dev deps do not include fastapi for tests/hermes_cli/test_dashboard_auth_401_reauth.py.
3. Rerunning the broad suite with that file ignored then aborted on the same missing fastapi dependency in tests/hermes_cli/test_dashboard_auth_cookies.py; CI should cover the full suite in the hermetic environment.
4. python scripts/check-windows-footguns.py gateway/run.py hermes_cli/auth.py tests/gateway/test_auth_fallback.py tests/hermes_cli/test_auth_codex_provider.py
5. ruff check gateway/run.py hermes_cli/auth.py tests/gateway/test_auth_fallback.py tests/hermes_cli/test_auth_codex_provider.py
6. git diff --check main...HEAD

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS / Darwin 24.6.0 arm64

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

• This skill is broadly useful to most users (if bundled) — see Contributing Guide
• SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
• No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
• I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

N/A

[liuhao1024]

Verified ✅

Reviewed the full diff across hermes_cli/auth.py, gateway/run.py, and 3 test files.

What I checked:

1. relogin_required flag discrimination — The gateway now uses getattr(auth_exc, "relogin_required", False) instead of is_rate_limited_auth_error() to classify AuthErrors. This is a cleaner signal: relogin_required=True means credential problem (auth failed), relogin_required=False means transient/quota (provider unavailable). The log messages correctly reflect the distinction. ✅

2. _pool_codex_rate_limit_remaining() early-return semantics — Returns None (not rate-limited) if ANY pool entry has a non-429 error code or an expired cooldown. This correctly handles mixed pools (429 + 401 = not all-rate-limited → auth remediation). The saw_token guard prevents false negatives on empty pools. ✅

3. Insertion point in resolve_codex_runtime_credentials — The cooldown check is placed AFTER the singleton check and BEFORE the read_error re-raise. This means: if a singleton token exists, it's used directly (no pool check). If no singleton, the pool is checked for all-429 before falling through to the generic "no credentials" error. ✅

4. Test coverage — 5 tests covering: all-pool-429 → rate-limit error, 401 pool → auth error, mixed 429+401 → auth error, transient log classification, credential log classification. The tests verify both code and relogin_required fields. ✅

No findings. The PR correctly separates quota exhaustion from credential failure in the Codex provider path, with proper fallback chain behavior.

[konsisumer]

Rebased onto origin/main and resolved the hermes_cli/auth.py conflict while keeping the PR's Codex pool cooldown behavior intact. ruff check, scripts/check-windows-footguns.py, git diff --check, and the PR-scoped tests (tests/gateway/test_auth_fallback.py, tests/hermes_cli/test_auth_codex_provider.py) passed locally. The bounded full-suite pytest tests/ -q -x --timeout=60 run still stops during collection on the pre-existing missing fastapi dev dependency in tests/hermes_cli/test_dashboard_auth_401_reauth.py.

[konsisumer]

Closing: referenced_issue_closed. all referenced issues closed: [32790]