fix(credential-pool): honor fuzzy throttle windows, stop cooldown bypass, wait for transient recovery

[someaka]

What this fixes

Real-world failure class observed against a 3-key custom-provider pool
(hyper.charm.land):

1. A transient upstream 429 with body "Please try again in a few minutes."
   benched the funded key for the full 1-hour default TTL — the fuzzy retry
   guidance was never parsed.
2. Rotation then walked into two genuinely-depleted 402 keys, the pool hit
   "no available entries", and the turn aborted as billing-exhausted — while
   the funded key recovered in minutes.
3. While the pool had keys benched, runtime resolution silently fell through
   to the config singleton key (custom_providers.api_key), serving keys the
   pool had benched for billing and burning requests on depleted accounts —
   the source of the "random" 402s.
4. Benches on config-seeded (borrowed) entries were additionally wiped on
   every load_pool(), because _upsert_entry treated secret re-hydration
   as a key rotation — so billing benches on the config key did not survive
   process restarts.

Changes

1. Cooldown sizing (agent/credential_pool.py):
   _extract_retry_delay_seconds now parses fuzzy retry guidance
   ("try again in a few minutes" → 180 s, "in N minutes" → N×60 capped at
   600 s, "in a moment/shortly" → 60 s). The parsed window rides
   error_context → reset_at → _exhausted_until, so the bench matches the
   provider's own guidance. Message-derived windows carry provenance
   (MESSAGE_DERIVED_RESET_TAG) and are dropped for billing-classified
   failures (preserving the HTTP 402 (payment required) incorrectly retried as transient error — causes runaway token spend #31273 money-burn protection) while
   provider-supplied structured reset_at fields/headers are kept.
2. Cooldown bypass (hermes_cli/runtime_provider.py):
   _try_resolve_from_custom_pool now reports pool exhaustion state and all
   three fallthrough call sites filter candidates through
   _pick_custom_api_key — billing-benched keys are never served. An
   all-benched pool raises AuthError(code="insufficient_credits"), which
   the gateway already routes to the fallback chain / billing-exhaustion UX.
3. Recoverable-exhaustion wait (agent/agent_runtime_helpers.py): when
   rotation reaches "no available entries" and the earliest benched entry is
   transient (not billing) and recovers within 600 s, the recovery path waits
   (interruptible, with activity heartbeat) and retries instead of aborting.
   Pure-billing exhaustion still aborts immediately. Auxiliary background
   calls are intentionally exempt (documented at _recover_provider_pool).
4. Fingerprint-aware rotation detection
   (agent/credential_pool.py::_upsert_entry): re-hydrating the same secret
   no longer clears persisted benches; a genuinely new key in config still
   does.

Relationship to existing work

• Fuzzy retry phrasing extends the gap documented in [Bug] credential_pool.py: Regex misses 'Please retry in X.Xs' format from Gemini API #51450 / fixed for
  numeric phrasings in fix(agent): parse Gemini retry-in cooldowns #51461 — non-numeric guidance ("a few minutes") is the
  delta here.
• The runtime_provider.py fallthrough site is the same bug class as [Bug] credential pool exhaustion causes misleading 401 instead of real 429/402 #40960
  / fix(auth): surface real upstream 429 when all pool entries exhausted (#40960) #57137 (auth.py path), fix(fallback): benched credentials must not disqualify a chain entry #79840 (fallback chain), and fix(auth): respect exhausted cooldowns in codex pool fallback resolver #43277 (codex
  resolver): an exhaustion verdict ignored on a fallthrough path. This PR
  closes the custom-provider runtime-resolution site.
• The recovery wait complements rather than replaces the TTL-shortening
  approach in fix(credential-pool): reduce TTL for HTTP 402 (Payment Required) from 1h to 2min #74946, and is adjacent to [Bug]: CLI/oneshot gets no resolution-time fallback — hermes -z dies for the whole quota window even with fallback_providers configured #81209/fix(oneshot): consult fallback_providers at resolution time #81517 and Recoverable 402 ("can only afford N tokens") is treated as terminal billing and drops the request #49769/fix(error): treat 402 'can only afford N tokens' as recoverable rate-limit #49785.
• The HTTP 402 (payment required) incorrectly retried as transient error — causes runaway token spend #31273 within-turn billing abort (money-burn protection) is preserved:
  billing-classified exhaustion never waits and still aborts;
  tests/run_agent/test_31273_402_not_retried.py passes.

Tests

• New: tests/agent/test_credential_pool_402_flapping.py — 36 cases covering
  fuzzy parsing, billing guards (incl. the full extractor → pool runtime
  seam), bench persistence across reload from disk, the resolution bench
  filter, wait-assessment semantics (incl. mixed transient/billing pools),
  and single-key/no-pool legacy behavior.
• Regression: 182 passed across the credential-pool and runtime-resolution
  suites. One pre-existing unrelated failure
  (test_qwen_oauth_auto_fallthrough_on_auth_failure) fails identically on
  the parent commit — qwen-oauth auto-resolution fallthrough, untouched by
  this diff.
• Perf: hot path unchanged (select() ≈ baseline); the exhaustion probe adds
  cost only in the all-benched state; config cache intact.
• Robustness: suite ×2 flake-free; 8-thread rotation storm ×3 (12,000 ops,
  zero torn writes); scripted flapping simulation 16/16; auth.json integrity
  39/39.

Scope note

The borrowed-entry persistence model is unchanged: _seed_custom_pool still
creates the config:<name> entry from custom_providers.api_key, and
borrowed entries persist without secrets by design (fail-closed disk
boundary). The fingerprint fix makes that boundary cooldown-correct; changing
the persistence model itself is a separate (security-sensitive) change.