fix(auth): a deliberate reset must outrank a binding cooldown

[rodrigogs]

Problem

hermes auth reset <provider> reports success and changes nothing — but only while the cooldown it is meant to clear is still in force.

Observed on a live install after topping up a DeepSeek account. The key was healthy (a direct POST /chat/completions with the pooled token answered 200, and /user/balance reported total_balance "19.92"), yet every agent invocation still refused the provider:

$ hermes auth reset deepseek
Reset status on 1 deepseek credentials

$ hermes auth list deepseek
deepseek (1 credentials):
  #1  api-key-1   api_key manual exhausted invalid_request_error (402) (21m 46s left)

$ hermes -z "..." -m deepseek-v4-flash --provider deepseek
hermes -z: agent failed: No usable credentials found for provider 'deepseek'.

Read back from auth.json in a fresh process immediately after the reset, last_status, last_status_at and last_error_code were byte-identical to their pre-reset values.

1. The concurrency guard cannot tell a reset from a stale snapshot. write_credential_pool merges on-disk status over the caller's entries via _merge_disk_cooldown_state, so one process cannot resurrect a key another has just benched. That merge adopts the disk copy when it is strictly more recent by last_status_at. reset_statuses clears last_status_at to None, which _parse_absolute_timestamp yields as 0.0 — older than any real timestamp. So a deliberate operator reset is by construction the losing side of that comparison, and the cooldown is copied straight back over the cleared fields.

2. The failure profile hides it. _merge_disk_cooldown_state returns early when the on-disk cooldown has already expired (until <= time.time()). So the command works whenever clearing was unnecessary, and silently does nothing whenever it was the reason you ran it. A test that resets an expired cooldown passes either way, which is why the existing coverage — test_query_method_acquires_lock, which only asserts that reset_statuses takes the lock — never saw this.

3. failure_reason was never cleared at all. It is persisted through _EXTRA_KEYS and lives in PooledCredential.extra, not as a dataclass field, so replace() could not reach it. Entries came out of a reset with no status and no error code but still classified billing, which hermes auth list renders as though it were current.

Fix

Thread the caller's intent through, mirroring the removed_ids parameter that already exists one concern over for the same reason — "do not resurrect this from disk, I removed it on purpose":

• write_credential_pool takes status_cleared_ids. Entries whose id is in that set skip _merge_disk_cooldown_state entirely. An id in the set means "I have seen the newer status and I am dropping it", as opposed to "I have not seen it".
• CredentialPool._persist forwards status_cleared_ids; reset_statuses passes the ids it cleared.
• reset_statuses also clears failure_reason (via extra) and treats its presence as a reason to clear, so an entry carrying only a stale classification is not skipped.

Every caller that does not declare the intent keeps the previous, protective behaviour unchanged.

Verification

• The reproduction above now clears on the first attempt, and has_available() is True from a fresh load_pool.
• Three new tests in tests/agent/test_credential_pool.py. The cooldown in the two regression tests is deliberately 5 seconds old, because an expired one passes with or without the fix:
  • test_reset_statuses_clears_a_cooldown_that_is_still_binding — asserts against what reached disk, not the in-memory object, which was correct all along.
  • test_reset_statuses_clears_the_classified_failure_reason
  • test_a_persist_without_declared_intent_still_cannot_erase_a_cooldown — the guard this fix is scoped against. Without it, "skip the merge always" would make the other two pass while reintroducing the lost update _merge_disk_cooldown_state exists to prevent.
• Mutation-checked: reverting only the two source files and keeping the tests fails the two regression tests and passes the guard test, so it is known to pin pre-existing behaviour rather than the new flag.
• pytest tests/agent/test_credential_pool.py tests/hermes_cli/test_auth*.py → 204 passed, 2 skipped. ruff clean on the changed files.

The wider -k "credential_pool or auth or fallback or failover" selection reports 27 failed on a clean main worktree (3638 passed) and the same 27 on this branch, with identical test names — in tests/tools/test_mcp_oauth.py, test_mcp_dashboard_oauth.py, test_terminal_tool_requirements.py, test_web_tools_config.py and tests/docker/. None of them import credential_pool or hermes_cli/auth.py. The three MCP OAuth ones fail in isolation too; the rest only inside the large selection, i.e. the order-dependent state leakage already noted in #79840.

Context

Same subsystem as #79840, which fixed a benched credential being mistaken for an unconfigured provider. That one made a cooldown recoverable automatically; this one makes it clearable deliberately.

[benperry6]

Confirmed on a live multi-account OpenAI Codex setup. Before this patch, hermes auth reset openai-codex reported that it reset five entries, but a fresh process immediately read the same binding 429 cooldowns again. One account had already regained quota after a plan upgrade; clearing only its status fields, without changing either OAuth token, allowed a real gpt-5.6-sol request to succeed.

I tested commit 7d13286 in an isolated runtime: the reset persisted across a fresh read, the account answered with gpt-5.6-sol, and the credential-pool tests I ran passed. This fixes a real recovery path for quota restored outside Hermes.

[rodrigogs]

Thanks for taking the time to reproduce this on a real multi-account setup — that
is the part I could not do from a single-account install, and it covers the exact
case the fix is about: quota restored outside Hermes, where nothing but the
stored status fields is stale.

Worth underlining one detail from your report for whoever gates this: you cleared
only the status fields and neither OAuth token changed. That is the whole point of
the status_cleared_ids parameter — it is the caller declaring "I have seen the
newer status and I am dropping it on purpose", so _merge_disk_cooldown_state
stops treating a deliberate reset as a stale snapshot. Nothing about the
credential material moves.

The branch is currently MERGEABLE with no conflicts against main; this repo
runs no CI on fork PRs, so there is no check to wait for on my side.

[rodrigogs]

Rebased onto current upstream/main (0a171fffe); new head 3a24755e4995.

No semantic drift. upstream/main has 40+ commits touching agent/credential_pool.py and
hermes_cli/auth.py since this branch's base — keyless providers, Bedrock/Vertex picker gating, UTF-8
auth-store reads, sole-credential cooldown work, billing-403 benching — and none of them altered the two
functions this PR modifies or the surrounding call shape: reset_statuses, _persist(removed_ids=...) and
write_credential_pool's signature are all unchanged, so the rebase was positional and the diff is
unchanged.

One cosmetic tidy is folded into the existing commit rather than added as a whitespace-only commit: the new
module-level helper _exhausted_billing_store in tests/agent/test_credential_pool.py was appended directly
against the preceding class body, so it now has the two blank lines PEP 8 wants. Ruff's configured rule set
does not enable E3xx, so nothing flagged it; it just read as jammed into the class above.

Verification on the new head: tests/agent/test_credential_pool.py → 62 passed (all 62 collected, nothing
deselected). One caveat on the local numbers: tests/agent/test_credential_pool_routing.py::TestCliTurnRoutePool::test_resolve_turn_includes_pool fails on this machine. It fails identically with a pristine upstream/main checkout at 0a171fffe, so it is pre-existing and unrelated to this change.

This repository does not run CI on pull requests from forks, so the checks tab stays empty and protect-main's required All required checks pass context never reports — which is why this PR shows mergeable: true with mergeStateStatus: BLOCKED. Approving the workflow run (or landing it on the strength of the local evidence) is all that is left from my side.

[benperry6]

A few repo-specific unblockers after the successful rebase, to avoid another CI round trip:

• @rodrigogs, could you enable Allow edits from maintainers on this PR? The API currently reports maintainerCanModify: false, which would prevent a maintainer from applying a small follow-up directly if review finds one.
• The contributor check will likely require an attribution mapping: the commit email is not currently mapped on main, and this PR changes only the three implementation/test files. The repository's supported fix is:

  python3 scripts/add_contributor.py rodrigo.smscom@gmail.com rodrigogs
  git add contributors
  git commit -m "chore: map contributor email"
  git push

• @teknium1 (or another maintainer covering area/auth), once that update is present, could you approve the current action_required workflow runs and review the PR? The current CI run is https://github.com/NousResearch/hermes-agent/actions/runs/32672550189.

The live multi-account OpenAI Codex evidence remains positive on the rebased semantics: the deliberate reset survives a fresh read, no OAuth token changes, and the recovered account completes a real gpt-5.6-sol request. The focused credential-pool tests also pass. This is the existing PR to land; no competing duplicate is needed.

[rodrigogs]

Thanks for the thorough live validation @benperry6 — the multi-account reproduction and the quota-restored recovery path are exactly the scenarios the fix targets.

Both unblockers are done:

• Allow edits from maintainers is now enabled on this PR (the API reported maintainer_can_modify: false before; it reads true as of now).
• Contributor attribution mapping added: ran scripts/add_contributor.py rodrigo.smscom@gmail.com rodrigogs and pushed the contributors/emails/ entry as a chore: map contributor email commit on top of fix(auth): make hermes auth reset actually clear a binding cooldown. The branch now contains only the implementation/tests plus that mapping commit.

The implementation itself is unchanged — hermes auth reset still clears the binding cooldown persistently (fresh-process read no longer resurrects it), with no OAuth token changes. Ready for the area/auth review whenever @teknium1 or a covering maintainer has a slot.