fix(dashboard-auth): coalesce concurrent refresh requests to prevent RT reuse detection

[liuhao1024]

What does this PR do?

Fixes a race condition in the dashboard auth middleware where concurrent requests after access-token expiry can trip refresh-token reuse detection, killing the remote dashboard session.

When the browser discovers an expired access-token cookie it fires a burst of parallel fetch() calls (session, profile, status, ws-ticket, etc.). Each carries the same old hermes_session_rt cookie. The first request rotates the RT and returns new cookies via Set-Cookie, but sibling in-flight requests still carry the old RT. Without coalescing, each replay triggers the provider's reuse-detection → RefreshExpiredError → session revoked → user kicked back to login.

Related Issue

Fixes #55712

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✅ Tests (adding or improving test coverage)

Changes Made

• hermes_cli/dashboard_auth/middleware.py — Add in-process replay cache (_refresh_cache) mapping sha256(old_refresh_token) to (new_session, provider_name, timestamp). An asyncio.Lock serializes concurrent refresh misses so at most one request per old-RT reaches the provider. Cache entries expire after 120 seconds. _attempt_refresh is now async to support the lock.
• tests/hermes_cli/test_dashboard_auth_middleware.py — Add ReuseDetectingProvider (raises RefreshExpiredError on second call with same RT) and test_refresh_token_replay_cache_prevents_reuse_detection regression test proving concurrent requests don't trip reuse detection.

How to Test

1. Run pytest tests/hermes_cli/test_dashboard_auth_middleware.py -x -q — all 34 tests should pass (33 existing + 1 new regression test).
2. The new test_refresh_token_replay_cache_prevents_reuse_detection specifically validates that two sequential requests with an expired AT and the same RT both succeed, and the provider's refresh_session is called exactly once (the second request hits the cache).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — N/A (server-side middleware, no platform-specific behavior)
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Log evidence from the issue reporter showing the problem:

{"event":"login_success","provider":"nous","ip":"192.168.64.117"}
{"event":"ws_ticket_minted","provider":"nous","ip":"192.168.64.117"}
...
{"event":"refresh_failure","provider":"nous","reason":"refresh_expired","ip":"192.168.64.117"}
{"event":"session_verify_failure","reason":"no_provider_recognises","ip":"192.168.64.117"}

The burst pattern (login_success → immediate refresh_failure) is the signature of concurrent requests replaying the same stale RT.

[teknium1]

Thanks for tracing the rotating-refresh-token path. The current-main premise remains: hermes_cli/dashboard_auth/middleware.py:352-388 refreshes each failed verification independently, while the Nous provider rotates RTs (plugins/dashboard_auth/nous/__init__.py:253-260).

Problems

• Blocking: hermes_cli/dashboard_auth/middleware.py:493 calls the synchronous provider refresh while holding the async lock. The Nous provider executes synchronous httpx.post at plugins/dashboard_auth/nous/__init__.py:269, so this blocks the ASGI event loop. This matches the #55712 follow-up, which specifically calls for threadpool execution.
• Blocking: tests/hermes_cli/test_dashboard_auth_middleware.py:671-691 captures rt_val but never uses it. r4 runs after r3 in the same TestClient, so this is neither concurrent nor an independently preserved stale-cookie request.
• hermes_cli/dashboard_auth/middleware.py:519 stores every successful result in a process-global dict, but expired entries are never removed.

Suggested changes

• Use per-token single-flight plus run_in_threadpool for synchronous provider refreshes.
• Test two isolated clients carrying the captured old RT concurrently, with a provider barrier/delay; also verify an unrelated endpoint remains responsive.
• Add bounded cache eviction and revisit the replay grace window.

Automated hermes-sweeper review.

[teknium1]

refresh_session is synchronous; the bundled Nous provider reaches synchronous httpx.post (plugins/dashboard_auth/nous/__init__.py:269). Calling it here blocks the ASGI event loop while this global lock is held, matching the #55712 recurrence report. Run it through starlette.concurrency.run_in_threadpool and coordinate only requests for this RT.

[teknium1]

Entries only become logically expired during lookup; none are removed from _refresh_cache. A long-running dashboard retains one Session and RT hash for every successful refresh until restart. Evict expired entries during insertion/access and bound the cache.

[teknium1]

This request is sequential and uses the same TestClient after r3 has processed Set-Cookie; the captured rt_val is never restored or supplied here. Use isolated clients/cookie snapshots and launch both requests concurrently against a delayed/barrier provider so the test exercises the stale-RT race.