feat: add provider rotation cooldowns

[vladimirdd96]

Summary

Adds persistent, quota-aware provider rotation so Hermes skips exhausted providers (rate-limit, billing) at the start of each new turn and across sessions.

Problem

The existing fallback_providers feature reacts to per-request failures but resets to the primary provider on every new turn. If the primary provider has hit its daily/monthly usage cap, every turn tries it first, fails, then falls back — wasting a round-trip and adding latency.

Solution

Introduce provider_rotation config (disabled by default) with a JSON state file (~/.hermes/provider_rotation_state.json) that persists cooldown records across turns and sessions. On a capacity failure the current provider is marked unavailable for a configurable window (6h for rate-limit, 24h for billing by default). Turn-start provider selection and fallback chain traversal both skip cooled providers.

Changes

• agent/provider_rotation.py — ProviderRotationState (load/save JSON), mark_unavailable(), is_unavailable(), reset(), filter_available_entries(), is_rotation_enabled(), cooldown_for_reason()
• hermes_cli/rotation_cmd.py — hermes rotation list/reset/clear CLI commands
• agent/chat_completion_helpers.py — mark provider on rate-limit/billing failure; skip cooled entries in fallback chain
• agent/agent_runtime_helpers.py — turn-start check: if primary is cooled and rotation enabled, advance to next available provider
• hermes_cli/config.py — provider_rotation config schema with defaults
• hermes_cli/main.py — wire rotation subcommand
• website/docs/user-guide/features/fallback-providers.md — document rotation vs fallback, config options, CLI

Config

provider_rotation:
  enabled: true
  cooldown_seconds: 21600              # 6h default
  cooldown_seconds_by_reason:
    rate_limit: 21600                  # 6h
    billing: 86400                     # 24h

Tests

31 new/updated tests — all pass via scripts/run_tests.sh:

• tests/agent/test_provider_rotation.py (3 tests)
• tests/hermes_cli/test_rotation_cmd.py (3 tests)
• tests/run_agent/test_provider_fallback.py (3 new rotation tests, 25 existing)

Backwards compatibility

• provider_rotation.enabled defaults to false — zero behavior change for existing installs
• Existing fallback_providers chain is preserved and reused as the rotation priority order

[alt-glitch]

Related: #22916 (same-provider auth-profile rotation on rate-limit/overload feature request). Also see merged #4188 (credential pools) and #12554 (fall back on rate limit when pool has no rotation room). This PR adds cross-session provider cooldown state, which is complementary to but distinct from credential pool rotation.

[teknium1]

Thanks for pursuing cross-session provider failover; the current 60-second per-agent fallback gate does not cover the persistent capacity case.

Problems

• agent/agent_runtime_helpers.py:854-884 only consults rotation state when _fallback_activated is false. After a normal fallback activation, once the existing 60-second gate expires, current restore logic proceeds to restore the primary instead of checking persistent state. This misses the long-lived CLI/gateway-agent case.
• agent/provider_rotation.py:27-31 keys only by provider/model. Current fallback identity includes base_url (agent/chat_completion_helpers.py:1301-1306), so separate custom endpoints sharing a provider/model would be cooled together.
• agent/provider_rotation.py:57-63 uses a shared fixed temporary pathname with no read-modify-write coordination, which can lose records when separate sessions update cooldowns concurrently.
• agent/chat_completion_helpers.py:753-765 treats every classified rate limit as a long cooldown. Current Nous handling requires evidence of an exhausted account bucket before writing its cross-session breaker (agent/nous_rate_guard.py:192-244); this generic path needs an equivalent distinction for transient/upstream 429s.

Suggested changes

• Apply persisted-primary selection on the restore path and add an expired-60-second/active-persisted-cooldown test.
• Include normalized endpoint identity in records and make state updates concurrency-safe.
• Gate durable cooldowns on provider evidence of durable exhaustion.

Automated hermes-sweeper review.

[teknium1]

This check only runs under if not agent._fallback_activated. After a normal fallback activation, the next long-lived-agent turn bypasses this block; once the existing 60-second _rate_limited_until expires, the code restores the primary without consulting this persisted cooldown. Please apply the check on the restore path too and cover that sequence.

[teknium1]

This key conflates distinct custom endpoints that share provider/model. Current fallback identity includes normalized base_url (agent/chat_completion_helpers.py:1301-1306) specifically to distinguish backend targets. Include the routing endpoint in cooldown identity and test two same-provider/model custom entries with different URLs.

[teknium1]

The shared cross-session state uses one fixed .tmp pathname and no read-modify-write coordination. Concurrent sessions can race on this path or overwrite each other's independent records. Use unique temporary files plus a lock or merge-on-write transaction.

[teknium1]

This records a multi-hour cooldown for every rate_limit. Current cross-session Nous handling only trips after proving an exhausted account bucket, to avoid suppressing healthy routes after transient/upstream 429s (agent/nous_rate_guard.py:192-244). Please add comparable durable-exhaustion evidence or provider-specific policy before persisting this state.