fix(auth): surface real upstream 429 when all pool entries exhausted (#40960)

[justdoit829]

Summary

Fix issue #40960: when all credential pool entries are in exhaustion cooldown (e.g. after a 429 quota-exhausted response), _resolve_api_key_provider_secret() silently returned an empty api_key. This caused the API request to fail with a misleading 401 Unauthorized, hiding the real cause (a 429 quota message from the provider).

Root Cause

hermes_cli/auth.py L593-605: when pool.peek() returns None (all entries in cooldown), the code returns an empty string for api_key instead of falling back to an exhausted entry.

Fix

After peek() returns None, iterate pool._entries and return the first usable key with a warning log. The upstream API then returns its real 429 with the quota-reset timestamp, giving users actionable information.

Test Plan

• Syntax check: ast.parse() passes
• Locally verified pool returns 401-masked-error before fix
• CI passes (GitHub Actions)
• Maintainer review

Related

• Closes [Bug] credential pool exhaustion causes misleading 401 instead of real 429/402 #40960
• See also Desktop model picker: credential pool exhaustion blocks provider switching after any 401 #52727 (related short-term fix proposal)

[alt-glitch]

This was generated by AI during triage.

Duplicate of #40961 - same fix in hermes_cli/auth.py (_resolve_api_key_provider_secret): fall through to an exhausted pool entry's key so the upstream returns its real 429 instead of a misleading 401. #40961 is the earliest open PR for #40960; #42475 is a sibling fix. Human should pick the canonical one from the cluster.

[teknium1]

Thanks for tracing the empty-key path. The premise is still present on current main: hermes_cli/auth.py:595-609 returns an empty key when pool.peek() has no available entry.

Problems

• The new loop at hermes_cli/auth.py:611 reads every raw entry, including STATUS_DEAD credentials. agent/credential_pool.py:1483-1516 deliberately excludes dead entries and entries inside exhaustion cooldowns; returning either can still produce the misleading 401 this change aims to avoid.
• Reissuing a request with a cooldown entry conflicts with the current recovery policy: run_agent.py:293-315 says retrying an exhausted quota burns retry budget, and the documented pool flow falls through to a configured fallback after all keys are exhausted (website/docs/user-guide/features/credential-pools.md:24-40).
• No regression test exercises peek() is None with exhausted/dead entries; the current fallback tests only cover a healthy pool entry (tests/tools/test_credential_pool_env_fallback.py:181-250).

Suggested changes

• Avoid raw _entries access; route the intended all-exhausted behavior through a pool API that preserves dead-entry and cooldown semantics.
• Add focused exhausted/dead-pool regression coverage after agreeing on the intended fallback or diagnostic behavior.

This is an automated hermes-sweeper review.

[teknium1]

This raw iteration includes STATUS_DEAD entries as well as cooldown entries. CredentialPool._available_entries() intentionally excludes dead credentials (agent/credential_pool.py:1483-1516); returning one here can send a permanently invalid key and retain the same 401 failure.