feat(auth): canonical shared xAI OAuth store for multi-profile single-use refresh tokens (#65394)

[KharonLaQua]

Summary

Fixes multi-profile death for xAI Grok OAuth (issue #65394): xAI issues single-use rotating refresh tokens, so when each Hermes profile forks its own copy into auth.json, the first profile to refresh revokes every other copy (invalid_grant). This PR introduces an opt-in canonical shared store that all profiles read and refresh under one shared lock — one grant, one refresher, no per-profile forking.

Activation is config.yaml (not a user-facing env var). This addresses the env-var-for-config closure of #67243 (AGENTS.md: non-secret feature flags belong in config.yaml; bridge to an internal env var if the mechanism needs one — same pattern as terminal.cwd → TERMINAL_CWD). The activation move leaves the core election/sweep/marker/lock engine unchanged; this refresh additionally hardens auxiliary shared-mode resolution so stale legacy pool entries can never bypass canonical or profile-disable state.

Problem

Today each profile (and the root ~/.hermes/auth.json) can hold its own xAI OAuth grant. xAI rotates the refresh token on every successful refresh:

1. Profile A refreshes → new RT written to A's store; old RT revoked at xAI.
2. Profile B still holds the old RT → next refresh gets invalid_grant.
3. Concurrent gateways / cron / desktop / CLI workers multiply the race.

This is the multi-profile fork/death path described in #65394. Write-through helpers and pool entries that also persist raw refresh tokens make the blast radius worse.

Solution

A canonical sole-owner store (not a Nous-style convenience layer) at:

${HERMES_SHARED_AUTH_DIR}/xai_oauth.json   # default: ~/.hermes/shared/xai_oauth.json
${HERMES_SHARED_AUTH_DIR}/xai_oauth.lock   # cross-process advisory lock

All profiles read / refresh / persist through one resolver under that lock. Profile and root stores keep only a non-secret source: shared:xai-oauth reference (no raw RT forking).

Opt-in gate (config.yaml user-facing; internal env bridge)

Shared xAI mode does not activate just because HERMES_SHARED_AUTH_DIR is set (that directory is already used for Nous shared auth).

User-facing activation (non-secret feature flag in config.yaml):

# ~/.hermes/config.yaml  (or a profile's config.yaml)
shared_auth:
  providers: [xai-oauth]
  # optional: dir: ~/.hermes/shared

hermes auth xai enable-shared    # writes the config key + bridges for this process
hermes auth xai disable-shared   # removes xai-oauth from shared_auth.providers

At process startup (CLI, gateway, cron, dashboard/serve, TUI), Hermes force-exports the internal bridge targets from that config (AGENTS.md env-var-for-config / terminal.cwd → TERMINAL_CWD precedent):

• HERMES_XAI_SHARED_AUTH=1 when providers includes an xAI alias
• HERMES_SHARED_AUTH_PROVIDERS=<comma-joined list>
• HERMES_SHARED_AUTH_DIR only when shared_auth.dir is set

The engine gate (_xai_shared_auth_enabled()) still reads only those env vars — they remain the internal mechanism so tests and power-user overrides keep working. Gate-off (absent/empty shared_auth in config) does not set or modify those env vars → byte-identical legacy behavior.

Key properties

Property	Behavior
Canonical sole-owner	One durable RT in the shared store; profiles never hold a forked secret copy
Atomic multi-store election + sweep	Under all-store locks: elect a winner, write shared, strip residual RTs across profile/root/pool/manual rows
Fail-closed shapes	Unreadable or wrong-shape stores (list, string, number) fail election / strip / sole-owner audit — never silent first-wins
Fleet sole-owner marker	Verifiable digest over inventory; stale/existence-only markers never skip re-audit
Fail-loud persistence	Durable write + parent-dir fsync; poison/partial write leaves non-promotable state
Quarantine compare-and-clear	Dead grants quarantined by generation; concurrent winner is adopted, not double-rotated
Full consumer routing	Runtime, credential pool, auxiliary client, proxy adapter, x_search / image / video / tts / stt, availability probes all go through the canonical resolver
Auxiliary canonical-first / fail-closed	In shared mode, auxiliary calls never consult a stale legacy pool before the canonical resolver; profile-disable, missing, or corrupt canonical state cannot fall back to legacy tokens
Codex / Nous isolation	Shared xAI path does not couple into Codex or Nous shared-auth ownership
Config-not-env activation	User-facing switch is shared_auth.providers; env vars are internal bridge targets only

Enablement + migration

# 1) Opt in (writes config.yaml; restart gateway/cron/desktop so they reload)
hermes auth xai enable-shared

# 2) One-time fleet migrate (or a single device-code login seeds the shared store)
hermes auth xai migrate-shared --source auto
# --source profile | --source root
# --force overwrites an existing shared grant

# 3) Login (if no legacy grant to migrate)
hermes auth add xai-oauth

Logout semantics under shared mode:

• hermes auth xai disable-shared — removes xai-oauth from shared_auth.providers in config.yaml (canonical grant stays on disk)
• hermes logout --provider xai-oauth — per-profile disable marker while shared mode remains on; canonical grant stays for other profiles
• hermes logout --provider xai-oauth --global — deletes the grant for every profile (intentionally noisy)

Requires a local filesystem with reliable advisory locking (not NFS/SMB). Every gateway/cron/desktop process must load the same config.yaml (shell-only exports are not the activation path).

Testing

The provider overlay was integrated and promoted as candidate
fce3427db0baabe316686d2bd3923cd20ddc4dbf, which contains upstream
07e97d2f5dc3d2092cfe693ef07b2527a36cd2d8. That merge completed without
textual conflicts. The same four commits in this PR have also been replayed
onto the upstream tip observed during this refresh,
32fd9d65cf091269709c5a6301b25aadac681aa8, without manual conflict
resolution. All 9 changed Python test files then passed: 131 passed.

The integration lane passed all of its normal gates:

• 229 passed, 2 deselected — exact historical provider/auth/runtime suite
• 330 passed — broader provider/auth/runtime superset
• 13 passed — focused semantic controls
• 33/33 passed — profile/model pin validation
• Scanner controls passed in both directions — the clean fixture was
  accepted and the positive fixture was detected

The pytest instruments were also tested against genuine zero collection. The
229-suite, 330-suite, and 13-control commands each returned pytest exit 5
when their selectors collected no tests. A green receipt from these gates
therefore cannot be a vacuous zero-test pass.

Deliberately poisoned candidates proved that the semantic gate detects and
names lost capabilities:

• dropping the auxiliary fail-closed commit reports
  auxiliary xAI shared credentials fail-closed
• dropping the config bridge reports shared-auth config bridge
• dropping the two-process canonical-store proof reports the canonical shared
  xAI store capability as missing

The historical 229-test suite alone still passed on a poisoned candidate.
That result is why the lane requires the semantic capability gate instead of
treating the historical suite as sufficient.

• tests/hermes_cli/test_shared_auth_config_bridge.py — config → env bridge, gate-off byte-identical, power-user env override preserved, enable/disable write config.yaml
• tests/hermes_cli/test_xai_shared_auth_store.py — gate off/on, generation bump, fail-loud persist, concurrent waiters adopt winner, quarantine compare-and-clear, migrate/strip sole-owner, election fail-closed on unreadable/wrong-shape stores, concurrent logout vs promote, fleet marker digest races, no-resurrection after quarantine, gate-off byte-identical legacy
• tests/hermes_cli/test_xai_shared_auth_two_process.py — genuine two-OS-process, cross-HERMES_HOME lock contention; one refresh winner and loser-adopts assertions. This directly exercises the OS-process boundary and real advisory locking raised in teknium1's review, rather than using threads as a proxy.
• tests/agent/test_auxiliary_xai_shared_recovery.py — aux auth-error recovery with rejected bearer / generation plus canonical-first, no-legacy-fallback regressions
• tests/agent/test_credential_pool_oauth_writethrough.py / test_credential_sources_xai_remove.py — pool write-through + source removal under shared mode
• tests/hermes_cli/test_xai_oauth_writethrough.py — OAuth write-through boundaries
• tests/tools/test_xai_http_shared_mode.py — tool HTTP path uses canonical resolver
• tests/plugins/video_gen/test_xai_plugin.py — plugin routing under shared mode

The gate-off path remains byte-identical to legacy behavior. Coverage also
includes deterministic concurrency (election→commit race, strip→inventory
race), fail-closed invalid shapes, and no-resurrection guarantees.

Production soak

The promoted change set has been running since 2026-07-25 in a real
multi-profile deployment spanning 8 gateways and 33 profiles. Live xAI and
Anthropic probes pass, and the xAI auth monitor reports
XAI_AUTH_HEALTHY with no affected profiles. This is operational soak
evidence from the promoted fork deployment; it is not a claim that upstream
CI has run or passed on this PR.

Docs

• website/docs/guides/xai-grok-oauth.md — Shared-store mode uses config.yaml / enable-shared
• website/docs/user-guide/configuration.md — shared_auth section
• website/docs/reference/environment-variables.md — HERMES_XAI_SHARED_AUTH / HERMES_SHARED_AUTH_PROVIDERS labeled internal bridge targets (do not set by hand)

Scope notes

• Opt-in only — default behavior unchanged until shared_auth.providers includes an xAI alias (or a power-user sets the internal env var).
• Activation does not alter the core engine gate — _xai_shared_auth_enabled() still reads env only; auxiliary shared-mode entry is additionally hardened to resolve canonical-first and fail closed.
• Does not change Codex or Nous shared-auth semantics beyond co-using the directory path convention.
• Closes / addresses [Feature]: Support shared xAI OAuth state across independent Hermes roots #65394; re-scopes activation to satisfy the policy that closed feat(auth): canonical shared xAI OAuth store for multi-profile single-use refresh tokens (#65394) #67243 (env-var-for-config).

Checklist

• Gate-off preserves legacy path (proven by bridge + store tests)
• User-facing activation is config.yaml (not a new HERMES_* .env flag)
• Internal env bridge at CLI / gateway / cron / dashboard / TUI startup
• All xAI OAuth consumers routed through canonical resolver
• Fail-closed election / strip / audit on bad store shapes
• Docs for enablement + migration + logout
• Adversarial concurrency + no-fork tests
• Two-process cross-profile acceptance test
• Auxiliary shared mode cannot bypass canonical/profile-disable state via stale legacy pool entries

[teknium1]

Thanks for re-scoping activation to config.yaml; the independent-root rotation premise is real on current main: xAI resolves the active store before only a same-install root fallback (hermes_cli/auth.py:4011-4015), and refresh serialization uses that local auth lock (hermes_cli/auth.py:4499).

Problems

• The new shared-store suite does not exercise the acceptance boundary from #65394. Its concurrent waiter test uses threading.Thread in one process (tests/hermes_cli/test_xai_shared_auth_store.py:272-339); it does not validate two OS processes with distinct HERMES_HOME values, one shared directory, and one refresh POST. That distinction matters for environment initialization, path resolution, and advisory locking.

Suggested changes

• Add the two-process regression described in the #65394 discussion: separate temporary roots, shared HERMES_SHARED_AUTH_DIR, a barrier before resolution, one mocked refresh endpoint, and assertions that the waiting process adopts the winner without retaining a local refresh-token fork.

Automated hermes-sweeper review.

[teknium1]

This concurrency test uses threads in one process. Please add a separate OS-process regression with distinct HERMES_HOME roots and one shared auth directory; the feature request specifically depends on cross-process path and advisory-lock behavior.

[KharonLaQua]

Addressed — thanks, this was the right call.

Added tests/hermes_cli/test_xai_shared_auth_two_process.py (commit 28826f9e1), which spawns two genuine child interpreters via subprocess.Popen with distinct HERMES_HOME roots pointing at one shared HERMES_SHARED_AUTH_DIR, so the cross-process path and the advisory lock are exercised for real rather than simulated with threads. The parent HERMES_HOME is explicitly popped from the child env so neither process can inherit it.

Key assertion is test_two_os_processes_shared_store_one_refresh_loser_adopts: exactly one process performs the refresh, the loser adopts the winner's result, and neither HERMES_HOME retains a durable local refresh-token fork.

The thread-based test you flagged is kept as the narrower in-process case rather than as the primary evidence.

Note this branch was rebased onto current main to clear merge conflicts, so your comment is showing as outdated — the commit is present in the new history.

[alt-glitch]

This was generated by AI during triage.

Related: #67243 is the closed predecessor and #65394 is the open shared-xAI-auth request. This resubmission changes activation to config.yaml but retains a broad cross-runtime shared-store migration, so it is related work rather than a duplicate.

[KharonLaQua]

Added the two-process regression you asked for: tests/hermes_cli/test_xai_shared_auth_two_process.py::test_two_os_processes_shared_store_one_refresh_loser_adopts (test-only commit). It exercises the actual #65394 acceptance boundary instead of in-process threads:

• Two real OS processes — subprocess.Popen(sys.executable, ...) × 2, each a separate interpreter with its own HERMES_HOME (rootA / rootB), both pointing at one shared HERMES_SHARED_AUTH_DIR. Asserts the worker PIDs are distinct and differ from the parent.
• Real kernel advisory locking — workers call the real resolve_xai_oauth_runtime_credentials; no monkeypatch of the shared lock, the resolve/adopt path, or refresh_xai_oauth_pure's HTTP body. The only patch is _xai_validate_oauth_endpoint, so a loopback stub URL is accepted (the *.x.ai HTTPS pin is unit-tested separately).
• One mocked refresh endpoint — a local single-use ThreadingHTTPServer: the first POST of the seeded RT returns rotated tokens and is held (so the winner keeps the flock while the loser queues on it); any later POST of the burned RT returns 400 invalid_grant.
• Barrier before resolution — a filesystem barrier releases both workers together, then coordination guarantees the loser is blocked on the advisory lock while the winner refreshes.

Assertions: exactly one successful refresh POST, invalid_grant count is zero (the loser adopts the winner's rotated token rather than re-POSTing the single-use RT), both processes end on the rotated access token, the shared store advances generation 1→2 with a single rotated RT, and neither HERMES_HOME retains a durable local refresh-token fork (only the source: shared:xai-oauth reference). 10/10 non-flaky locally.

The existing threaded test stays as a fast in-process check; this adds the real cross-process boundary. Thanks for the catch.

[KharonLaQua]

Updated 2026-07-24:

• Rebased the PR onto current origin/main (7cd48733d).
• Added the exact two-OS-process / distinct-HERMES_HOME acceptance boundary requested in the sweeper review: 2b6e37731 (tests/hermes_cli/test_xai_shared_auth_two_process.py). It uses one shared auth directory, real cross-process advisory-lock contention, a single-use refresh stub, and asserts the loser adopts the winner without retaining a local refresh-token fork.
• Added auxiliary canonical-first / fail-closed hardening in a723ddee4, preventing stale legacy pool credentials from bypassing canonical or profile-disable state.
• Verification on current upstream: 131/131 PR-changed tests passed; 372/372 surrounding xAI OAuth, migration, provider-resolution, and runtime tests passed.

The requested acceptance gap is addressed. Please re-review the refreshed head 49207792e.

[KharonLaQua]

Refreshed the branch and re-verified its four commits against today's current upstream tip 32fd9d65 (the promoted production candidate includes upstream 07e97d2f): the replay was conflict-free and all 131 changed-file tests passed. test_xai_shared_auth_two_process.py uses two real OS processes with separate HERMES_HOME values and real advisory-lock contention, directly addressing teknium1's earlier point that threading tests did not prove that boundary. The change set has been running since 2026-07-25 in a live multi-profile deployment across 8 gateways and 33 profiles, with live xAI and Anthropic probes passing. The PR currently reports no checks; if CI for fork PRs requires maintainer approval, I would appreciate having it triggered.

[GottZ]

This was generated by AI during triage.

Summary

Seven PRs address the same xAI single-use refresh-token race across profiles. #28375 and #44604 route xAI state through the root auth store, #33284/#37243/#47290 introduce variants of a separate shared store, #67243 proposes a broad canonical sole-owner store but violates the repository’s configuration policy, and #67261 is its config.yaml-based resubmission, although no diff is available here to verify its claimed acceptance-gap fixes.

Related pull requests

• #28375 [closed] related — (+280/-60) — implemented on main: routes xAI OAuth reads, rotations, locking, and credential-pool synchronization through the global root auth store, directly preventing profile-local copies from racing; it remains relevant as the closed reference superseded by merged #46614.
• #33284 [closed] related — (+683/-3) — duplicate of #28375: serializes refreshes through a separate shared xAI store and merges rotated tokens into profile state, addressing the same race but adding parallel storage infrastructure; it remains relevant as a closed alternative whose author accepted consolidation on #28375.
• #37243 [closed] related — (+358/-25) — duplicate of #28375: redirects named-profile reads, writes, refreshes, and credential-pool synchronization to a shared xAI store while preserving classic mode; it remains relevant because its author closed it after accepting #28375 as the broader consolidation target.
• #44604 [closed] related — (+290/-30) — prior cleanly reviewed root-store implementation: makes global xAI provider and pool state canonical for profiles, uses path-scoped locks, and quarantines terminal failures in the global store; it remains relevant as a closed reference, but its narrower root-auth design overlaps behavior already implemented on main.
• #47290 [closed] related — (+415/-61) — duplicate shared-store variant: moves named profiles to shared/xai_oauth.json, removes profile token copies, and routes credential-pool synchronization through that store; it remains relevant as a closed implementation of substantially the same change as #37243.
• #67243 [closed] related — (+7787/-191) — superseded predecessor of #67261: implements a broad canonical sole-owner store across runtime, pool, proxy, tools, migration, logout, and recovery paths, but the blocking contributor review rejected its user-facing HERMES_* activation because repository policy requires config.yaml settings.
• #67261 related — (+9030/-191) — keep open pending refreshed maintainer verification: the PR description says it re-scopes #67243 activation to config.yaml, and the author reports adding the requested two-process regression, but no #67261 diff is available here to verify the claimed one-refresh, loser-adoption, or no-local-fork behavior. This follows the visible keep_open/salvageability=medium review rather than treating author comments as proof that its acceptance gap is resolved.

Duplicates

#33284, #37243, and #47290 are substantially duplicate shared-store fixes; #28375 and #44604 substantially overlap on making the global root auth store canonical. #67243 is the closed predecessor of #67261, not an independent merge candidate.

Suggested consolidation

Do not merge #67261 yet; keep it open and request a refreshed maintainer review against its current diff and executed two-process acceptance test. The visible keep_open/salvageability=medium review identified a missing cross-process boundary, and the available data contains only author claims—not a #67261 diff or refreshed verification verdict—saying that gap is closed. #33284, #37243, and #47290 can remain closed as duplicates, #67243 remains superseded by #67261 due to its configuration-policy conflict, and #28375/#44604 remain closed references because the root-store behavior is already represented on main via #46614.

Cross-PR triage: Reviewed 7 pull requests and 0 issues in this complex. Diffs were read for 6 of 7 PRs (rest unavailable); Assessment working set: 481 kB of PR diffs, 26 kB of issue/PR text, 14 kB of discussion (16 comments), 4 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.