fix(xai-oauth): serialize refreshes across profiles

[alexgunsberg]

Problem

xAI rotates OAuth refresh tokens after each use. In profile mode, Hermes previously serialized refreshes only on each profile's own auth.lock. Two profiles could therefore hold different locks while spending the same global-root fallback token. A parallel worker burst made one refresh succeed and the others replay the consumed token, permanently returning invalid_grant / Refresh token has been revoked.

The credential-pool path had the same cross-profile gap: each profile's local pool mirror could remain stale even after another profile rotated the root singleton.

Fix

• Hold the active profile auth lock and the distinct global-root auth lock across the complete xAI re-read → refresh POST → persist transaction.
• Preserve the established profile-before-root lock order for rolling-upgrade safety.
• For singleton-seeded xAI pool entries, re-read the authoritative auth-store singleton while those locks are held instead of trusting a profile-local pool mirror.
• Keep manual pool entries on their existing exact-row sync path.

Regression coverage

The new threaded regression creates two profile auth stores sharing one root xAI grant and starts simultaneous refreshes through both runtime paths:

• direct resolve_xai_oauth_runtime_credentials()
• credential-pool load_pool("xai-oauth").select()

Before the fix, both tests observe two refresh POSTs. After the fix, exactly one refresh occurs and both profiles receive the rotated access token.

Verification

• 61 passed across the new regression and existing xAI auth/pool/recovery suites
• ruff check passed on changed files
• compileall passed on changed files

[alexgunsberg]

Maintainer request: this PR (and several sibling fork PRs from the same contributor) have statusCheckRollup: [] — GitHub Actions has never run, which looks like the fork workflow-approval gate rather than red CI.

Could a NousResearch maintainer please click Approve and run workflows on this PR (and, if convenient, the related open PRs from @alexgunsberg)? Happy to rebase anything that has gone stale once checks can run.

Related open PRs currently also showing zero checks: #81802, #81927, #82508, #82537, #82607, #82608, #83155, #83181, #83183, #83337.

[johnmwhitman]

Independent confirmation of this race from a production multi-profile deployment (10 profile gateways, per-profile HERMES_HOME, shared root grant): repeated xai-oauth grants died within minutes of hermes auth add with invalid_grant — xAI rotates the refresh token on every use, and two profiles each holding only their own profile-store lock can race the same root-resolved token; the loser replays a consumed token and xAI revokes the family. We lost 11 consecutive grants to this before locking refreshes across profiles (we serialize on the global root store's lock; this PR's profile-then-root ordering is stricter and looks right). Can confirm the failure mode and that serialization fixes it — after applying an equivalent lock, grants have survived multi-day multi-profile use with zero invalid_grant recurrences.

One adjacent bug worth a look while in this code: on terminal refresh failure the quarantine path writes the emptied provider state into the active profile's store, which creates a shadowing providers.xai-oauth stub that permanently hides the root grant from that profile (the #74339 shape). Happy to file that separately.