Skip to content

fix(credential-pool): back off repeated exhaustion - #73750

Closed
konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/credential-pool-exponential-backoff
Closed

konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/credential-pool-exponential-backoff

Conversation

@konsisumer

Copy link
Copy Markdown
Contributor

Fixes #15296

What changed and why

Per the reporter follow-up on 2026-07-28, credential-pool entries now persist a defensive consecutive_failures counter and use a capped exponential cooldown (1h → 2h → 4h → 8h for 429/402-style failures). Automatic expiry makes an entry eligible for another request without clearing its streak, so another exhaustion extends the cooldown instead of returning to the flat base TTL. Fresh-token synchronization, a successful token refresh, and hermes auth reset clear the streak.

Addressing maintainer feedback

The related-work review named #15455 (closed) and #43856 (open). This new branch carries forward #15455's required lifecycle behavior—preserving the streak across normal cooldown expiry—and avoids #43856's premature reset during expired-entry selection. It also resets the streak during Anthropic fresh-token synchronization.

How to test

  • /opt/homebrew/bin/timeout -k 30 480 sh -c 'pytest tests/agent/test_credential_pool.py -q -x --timeout=60 "$@"' sh — passes (111 tests).
  • /opt/homebrew/bin/timeout -k 30 480 sh -c 'pytest tests/ -q -x --timeout=60 "$@"' sh — attempted, but collection is blocked by the environment's missing fastapi dependency for dashboard tests; no dependencies were installed.

What platforms tested on

  • macOS (Darwin arm64) local worker environment.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/auth Authentication, OAuth, credential pools P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jul 29, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Duplicate of #43856: both open patches implement the same credential-pool consecutive-failure exponential backoff and conflict on the same code path. #43856 is the earlier active patch.

@konsisumer

Copy link
Copy Markdown
Contributor Author

Closing in favour of #43856, which covers the same ground and predates this PR by seven weeks.

Both patches add backoff to agent/credential_pool.py for repeated credential exhaustion. #43856 is the earlier and broader change (+297/-7 across three files, including hermes_cli coverage), so there is no reason for this one to compete with it — the duplicate call in the triage comment above is correct.

Nothing here is worth salvaging separately as far as I can tell; if any detail of this diff turns out to be missing from #43856, it is easier to raise it there than to keep a second patch open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/auth Authentication, OAuth, credential pools comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Credential pool: no exponential backoff on repeated exhaustion — flat TTL causes 429 retry loops

2 participants