fix(anthropic): serialize Claude Code OAuth refresh under the auth-store lock

[loop2zero]

Summary

Native Anthropic (model.provider: anthropic, Claude Code OAuth) silently falls back to another provider under concurrent load, even when the Claude subscription is healthy and claude -p --model ... works directly. Every gateway/Discord turn logs:

Primary provider auth failed: No Anthropic credentials found ... — trying fallback
Fallback provider resolved: openai-codex model=gpt-5.5
Runtime provider supplied explicit model override: claude-opus-4-8 -> gpt-5.5

and hermes auth list anthropic shows the sole claude_code entry as rate-limited (429).

Root cause

Claude Code OAuth refresh tokens are single-use. Multiple concurrent Hermes turns (several gateway sessions, delegation subagents, cron jobs) each POST the same refresh token to Anthropic's token endpoint, invalidate one another, and drive the endpoint into 429s. With a single Anthropic credential in the pool, the entry is then marked exhausted and every live turn routes to the fallback provider — silently, for up to the cooldown TTL.

This is the same single-use-refresh-token concurrency race that #56233 fixed for Codex:

"...two concurrent Hermes processes could both adopt the same on-disk token and both POST it; the loser got refresh_token_reused / invalid_grant."

That fix wrapped the Codex branch of _refresh_entry in the shared _auth_store_lock but was deliberately scoped narrow ("this ships the narrow lock-serialization fix") and never generalized. Anthropic Claude Code OAuth has two refresh paths with the same shape, neither serialized:

Refresh path	Codex	Anthropic (before this PR)
Credential-pool _refresh_entry	✅ locked (#56233)	❌ unlocked
Direct _refresh_oauth_token	(no independent path)	❌ unlocked since introduction

Why the asymmetry existed:

1. anthropic_adapter._refresh_oauth_token predates the credential pool. The direct adapter refresh landed 2026-03-12; credential_pool.py (which owns _auth_store_lock) landed 2026-03-31. The direct path was written around Claude Code's own ~/.claude/.credentials.json file flow and never learned about the later auth-store lock.
2. Codex and Anthropic use different backing stores. Codex tokens live in auth.json (providers.openai-codex), already under _auth_store_lock, so "wrap it in the existing lock" was the natural Codex fix. Anthropic claude_code tokens live in Claude Code's ~/.claude/.credentials.json, historically guarded only by a file-sync mechanism (_sync_anthropic_entry_from_credentials_file) — which narrows the race window but cannot eliminate it for a single-use token.
3. fix(auth): serialize Codex OAuth pool refresh under the auth-store lock #56233 was intentionally Codex-only and the structurally-identical Anthropic pool path was not carried along.

Changes

Both commits reuse the exact pattern established by #56233 — the existing reentrant, cross-process _auth_store_lock, with an in-lock re-read so a waiter adopts the token the winner rotated instead of re-POSTing a consumed one.

Commit 1 — pool refresh path (agent/credential_pool.py)
Wrap the provider == "anthropic" and source == "claude_code" branch of _refresh_entry in _auth_store_lock, re-syncing from the credentials file inside the lock. Directly mirrors the Codex block added in #56233.

Commit 2 — direct refresh path (agent/anthropic_adapter.py)
Serialize _refresh_oauth_token's read→POST→write-back under the same reentrant lock, with an in-lock re-read that adopts a concurrently-rotated token and skips the POST. This path is hit on every anthropic_messages API call via _try_refresh_anthropic_client_credentials() and by auxiliary_client refresh, so it covers the main agent, delegation subagents, and cron jobs. Falls back to the unserialized path only if the lock helper is unavailable.

Together these cover all Claude Code OAuth refresh entry points.

Reproduction

1. Configure native Anthropic via Claude Code OAuth (single pooled claude_code credential).
2. Drive several concurrent turns (e.g. multiple Discord threads + a delegation batch + a cron job firing in the same tick).
3. Observe: auth list anthropic flips to rate-limited (429), and every subsequent turn logs Primary provider auth failed ... Fallback provider resolved: openai-codex, running Claude-configured sessions on the fallback model instead.

Testing

• pytest tests/agent/test_credential_pool.py — 87 passed (incl. 2 new regressions: refresh runs under the lock; a resync short-circuits the POST).
• pytest tests/agent/test_anthropic_adapter.py — 175 passed with isolated HOME (incl. 2 new regressions: direct refresh runs under the lock; a concurrently-rotated token is adopted without re-POSTing).
• Combined on a clean checkout of this branch off main: 262 passed.

Verified live on a running gateway after applying both commits: main agent, delegation subagents (platform=subagent), and an agent-mode cron job all log provider=anthropic with zero Fallback provider resolved / Runtime provider supplied explicit model override lines, and the pooled credential stays active instead of flipping to rate-limited.

Related

• fix(auth): serialize Codex OAuth pool refresh under the auth-store lock #56233 — the Codex-only precedent this generalizes to Anthropic.

[teknium1]

Thanks for tracing both current refresh paths; the underlying race is present on current main (agent/anthropic_adapter.py:1097-1123, agent/credential_pool.py:962-1022).

Problems

• The proposed lock is profile-local, not Claude-Code-store-local. _auth_store_lock() locks get_hermes_home()/auth.json (hermes_cli/auth.py:895, :983-984), but the token being refreshed is shared at Path.home()/.claude/.credentials.json (agent/anthropic_adapter.py:934, :1143). Two profiles can hold distinct auth-store locks and still POST the same shared refresh token. The new tests use one HERMES_HOME, so they do not cover this path.
• The diff adds HERMES_ANTHROPIC_REFRESH_TIMEOUT_SECONDS; AGENTS.md prohibits new user-facing non-secret HERMES_* configuration. Use config.yaml if this needs to be configurable, or retain a fixed internal timeout.

Suggested changes

• Use a dedicated cross-profile lock for the shared Claude Code credential store and add a two-profile/single-HOME concurrency regression proving one POST.

Automated hermes-sweeper review.