fix(cron): fallback providers when OAuth token refresh returns 429

[immuhammadfurqan]

Summary

Fixes #46511 — cron jobs using OAuth providers (e.g. openai-codex) crashed with RuntimeError when the credential pool was exhausted and the primary provider resolution failed with a non-AuthError exception (e.g. HTTP 429 from the token refresh endpoint), instead of using the configured fallback_providers chain.

• Primary fix: The except Exception handler in cron/scheduler.py now walks fallback_providers/fallback_model before raising RuntimeError, mirroring the existing AuthError handler above it.
• Secondary fix: pool.has_available() guards the credential_pool assignment so a fully-exhausted pool (entries exist but all on cooldown) is no longer forwarded to AIAgent, allowing the agent-level fallback chain to fire cleanly.

Root cause

resolve_codex_runtime_credentials() reads an expired OAuth access token, attempts refresh_codex_oauth_pure(), and the OpenAI token refresh endpoint returns HTTP 429. This raises a plain Exception (not AuthError). The except AuthError handler already had correct fallback logic; the except Exception handler did not.

Test plan

• test_non_auth_exception_uses_fallback_providers — primary raises plain Exception, job succeeds via fallback_providers
• test_non_auth_exception_no_fallback_raises — no fallback configured, job fails cleanly
• test_all_fallbacks_exhausted_raises — all fallback entries also fail, job fails cleanly
• test_exhausted_pool_not_passed_to_agent — has_available()=False results in credential_pool=None in AIAgent kwargs
• test_available_pool_is_passed_to_agent — has_available()=True pool is forwarded as expected

pytest tests/cron/test_scheduler.py  # 141 passed

[liuhao1024]

Code Review — clean ✅

Reviewed the full diff (2 files, +222/-9). Findings:

1. Fallback chain is well-structured. The except Exception handler now mirrors the existing AuthError handler's fallback pattern — correct fix for Cron jobs don't fallback when credential pool is exhausted (OAuth providers) #46511 (OAuth 429 during token refresh wasn't consulting fallback_providers).

2. Credential pool exhaustion guard is correct. The has_available() check prevents forwarding an exhausted pool to AIAgent, which would otherwise pick a dead entry and fail immediately instead of letting the agent's own fallback chain kick in.

3. No double-fallback risk. The fallback_model variable read at line 1710 (existing code) handles model-level fallback within a resolved provider. The new block handles provider-level fallback when resolution itself fails. These are orthogonal — no collision.

4. Test coverage is solid. Four tests cover: non-auth exception with fallback, no-fallback path, all-fallbacks-exhausted path, and the exhausted-pool guard. Edge case for has_credentials()=True but has_available()=False is explicitly tested.

No issues found. The fix is well-scoped and the tests document the intended behavior clearly.

[teknium1]

Thanks for the focused cron investigation. This is already implemented on current main; closing as an automated hermes-sweeper review.

• Codex OAuth refresh HTTP 429 is now raised as AuthError(code=CODEX_RATE_LIMITED_CODE) in hermes_cli/auth.py:3460-3482. Cron catches that error and resolves each configured fallback via get_fallback_chain(_cfg) in cron/scheduler.py:2930-2948.
• The reported runtime exhausted-pool path is handled by the shared agent loop: agent/conversation_loop.py:3193-3228 invokes the rotation-room check, and run_agent.py:291-313 returns false for a single-entry or fully exhausted pool so _try_activate_fallback() runs.
• This behavior was shipped by 1fc77f995ba892b675f7120cf29fcd2d90add31c (fix(agent): fall back on rate limit when pool has no rotation room) and is covered by tests/run_agent/test_provider_fallback.py:226-251.
• The PR is now conflicting with current main, and its added scheduler fallback branch would duplicate behavior already provided by the AuthError and shared runtime-fallback paths.