fix(auth): canonicalize inherited Codex refresh state

[Thomisso]

What does this PR do?

Makes the default/root Hermes auth store the canonical source for inherited openai-codex OAuth state across named profiles.

Codex refresh tokens rotate and are single-use. A named profile could previously read an inherited root grant, refresh it while locking or persisting the active profile store, and leave root or another in-memory pool snapshot holding stale state. That can cause refresh_token_reused, recreate local profile shadows, or roll a separately refreshed pool row backward.

This ports and completes the canonical-root direction proposed in #28277 on current main, including the credential-pool path requested by its review. It complements #41705 without scanning sibling profile stores, and extends the root write-through landed in #52760 with canonical ownership, locking, and stale-snapshot-safe persistence.

Related Issue

Fixes #6653

Related: #28277, #41705, #52760

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✅ Tests (adding or improving test coverage)

Changes Made

• hermes_cli/auth.py
  • preserve the exact auth-store source used by inherited Codex singleton and pool reads
  • lock and persist refreshes against that canonical source path
  • make nested file-lock reentrancy path-scoped so profile and root locks are never conflated
  • add source-aware credential-pool reads/writes
  • perform a baseline-aware three-way field merge under the canonical lock, preserving independent on-disk token rotations while applying local status/priority changes and explicit removals
  • allow direct Codex resolution to consume root pool-only credentials
• agent/credential_pool.py
  • carry the canonical Codex auth source path through pool lifetime
  • re-read the exact pool row after taking the canonical lock before spending a rotating refresh token
  • persist refresh, status, quarantine, and removal changes to the canonical store without creating profile-local shadows
  • adopt the merged canonical rows after writes so subsequent mutations use a current baseline
• Regression tests cover inherited singleton and pool refresh, malformed-token quarantine, same-row and different-row stale snapshots, concurrent rotation plus removal/reprioritization, root pool-only resolution, path-scoped locks, and profile-shadow prevention.

How to Test

python -m pytest tests/hermes_cli/test_auth*.py tests/agent/test_credential_pool*.py -q -o 'addopts='
# 449 passed

python -m ruff check hermes_cli/auth.py agent/credential_pool.py \
  tests/hermes_cli/test_auth_profile_fallback.py \
  tests/hermes_cli/test_auth_codex_provider.py \
  tests/agent/test_credential_pool_oauth_writethrough.py
# All checks passed

python scripts/check-windows-footguns.py hermes_cli/auth.py agent/credential_pool.py \
  tests/hermes_cli/test_auth_profile_fallback.py \
  tests/hermes_cli/test_auth_codex_provider.py \
  tests/agent/test_credential_pool_oauth_writethrough.py
# No Windows footguns found

git diff --check origin/main...HEAD

A live Linux/ARM64 multi-profile deployment was also migrated to one root Codex pool. Five independent profile invocations returned successful Codex responses, named-profile stores remained shadow-free after the calls, and all persistent gateways remained healthy.

The full repository suite was attempted on the ARM64 Raspberry Pi host but was stopped at 16% after roughly nine minutes because it was not practical to complete there. CI should provide the complete repository result; the entire affected auth and credential-pool suite passed.

Checklist

Code

• I've read the Contributing Guide
• My commit message follows Conventional Commits
• I searched open and closed related PRs/issues
• My PR contains only related changes
• I've run the entire pytest tests/ -q suite locally (affected suite: 449 passed; full suite deferred to CI)
• I've added regression tests
• Tested on Linux ARM64 (Raspberry Pi)

Documentation & Housekeeping

• Documentation update: N/A; internal auth persistence semantics, no new user-facing configuration
• cli-config.yaml.example: N/A; no config keys added
• CONTRIBUTING.md / AGENTS.md: N/A; no contributor workflow change
• Cross-platform impact considered; platform-neutral file/path logic and Windows-footgun checks pass
• Tool descriptions/schemas: N/A

Design lineage

Credit to #28277 for proposing the canonical-root Codex store direction. This PR is a current-main completion of that approach across both singleton and credential-pool paths, incorporating the maintainer review request to retain current pool behavior and avoid sibling-store freshness heuristics.

[teknium1]

Thanks for carrying the canonical-source approach through both Codex singleton and credential-pool paths.

Current main still has the reported ownership gap: inherited singleton resolution reads through _load_provider_state() after locking the active store (hermes_cli/auth.py:3238-3243), while _save_codex_tokens() persists through that active store (hermes_cli/auth.py:3386-3407). The pool path likewise falls back to root (hermes_cli/auth.py:1361-1366) but persists to the active store (hermes_cli/auth.py:1389-1417); the existing regression even asserts the resulting profile copy at tests/agent/test_credential_pool_oauth_writethrough.py:103-106.

The PR preserves the accepted root-fallback model while making Codex ownership explicit (hermes_cli/auth.py:1381-1431 on PR head), targets writes and locks to that source, and adds coverage for root-only refresh, pool-row reload, removal, and stale-snapshot merging. The PR base is present locally, and current post-base auth changes are limited to documentation commit 5fc2d9e64, so this appears suitable for straightforward salvage.

Automated hermes-sweeper review.

[solovision24]

Halo validation: PR #64572 is still OPEN and CONFLICTING with main. I attempted the supported GraphQL updatePullRequestBranch(REBASE) mutation, but the authenticated account solovision24 is forbidden from updating Thomisso's cross-repository branch (maintainerCanModify is true, but this token lacks that permission). Please rebase fix/codex-canonical-profile-auth onto current main, then I can rerun the focused auth/pool suites and close out the Kanban task.

[GottZ]

This was generated by AI during triage.

Summary

Thirty-seven PRs address or reference this Codex OAuth/auth-store complex. Their diffs span CLI-token isolation and recovery, profile/global pool fallback, rotating-token serialization and root write-through, sibling-profile recovery, usage lookup, diagnostics, and adjacent credential-pool concurrency fixes; the remaining #6653 ownership gap is most comprehensively targeted by #64572.

Related pull requests

• fix(auth): actionable error message when Codex refresh token is reused by another client #5612 [merged] partial — (+19/-5) — n/a: Added actionable diagnostics for reused Codex refresh tokens and Codex-specific HTTP 401 guidance; merged as an early diagnostic reference, not a state-coherence fix.
• fix: write refreshed Codex tokens back to ~/.codex/auth.json #8277 [merged] related — (+164/-2) — n/a: Wrote rotated tokens back to ~/.codex/auth.json across singleton and pool refresh paths; merged historical approach later reversed by fix(codex): Hermes owns its own Codex auth; stop touching ~/.codex/auth.json #12360's client-isolation design.
• fix(auth): parse OpenAI nested error shape in Codex token refresh #11512 [closed] partial — (+134/-6) — n/a: Parsed OpenAI's nested refresh error shape so refresh_token_reused reaches the existing reauthentication path; closed after its implementation and tests landed through fix(codex): consolidated OAuth error parsing + failed-status fallback routing + reauth UX #15104.
• fix(codex): Hermes owns its own Codex auth; stop touching ~/.codex/auth.json #12360 [merged] best fix — (+9/-282) — n/a: Removed runtime import, synchronization, and writeback involving ~/.codex/auth.json so Hermes owns separate Codex auth; merged reference implementation for issues Codex auth integration consumes shared refresh tokens from ~/.codex and breaks other clients #7806 and Codex OAuth refresh token gets consumed by sibling Codex clients (Codex CLI / VS Code extension), permanently invalidating Hermes session until manual re-auth #22903.
• fix(codex): consolidated OAuth error parsing + failed-status fallback routing + reauth UX #15104 [merged] partial — (+407/-65) — n/a: Consolidated nested OAuth error parsing, fallback routing, and explicit reauthentication UX; merged reference preserving fix(auth): parse OpenAI nested error shape in Codex token refresh #11512 and related work.
• fix(auth/gateway): surface Codex OAuth credential from credential_pool to /usage (#15167) #15173 [closed] best fix — (+417/-1) — n/a: Added pool-backed Codex token lookup and cold-start /usage provider probing; closed as superseded by the smaller, rebased fix(auth): resolve Codex usage credentials from auth fallbacks #17480.
• fix(auth): resolve Codex usage credentials from auth fallbacks #17480 [closed] best fix — (+213/-3) — n/a: Added global/pool-aware token resolution and cold-start /usage probing, but projected independent pool credentials through singleton refresh state; closed at the author's request despite the keep_open review, which required pool-native refresh handling.
• fix(auth): share Codex OAuth across profiles #28277 partial — (+96/-13) — n/a: Canonicalized only the Codex singleton in the global root; keep-open review found the pool path uncovered and the save implementation stale, making this a salvage source rather than a direct apply.
• fix(auth): honor shared auth home for profiled workers #29502 [closed] partial — (+118/-8) — n/a: Introduced HERMES_AUTH_HOME as an explicit shared auth root for profiled workers; closed after conversion to issue Profiled workers need a shared auth home separate from isolated HERMES_HOME #29530 and overlaps fix(auth): share Codex OAuth across profiles #28277/feat(auth): HERMES_AUTH_FILE env var to share auth.json across profiles #20405.
• fix(auth): resolve Codex credentials from pool fallback #29873 [closed] fixes — (+49/-1) — n/a: Added singleton-reader fallback to a pooled Codex credential; closed because broader upstream handling landed in 69dfcdc.
• fix(auth): share Codex OAuth pool across profiles #34121 [closed] partial — (+389/-42) — n/a: Shared Codex pools at global root and synchronized device-code entries; closed as implemented on main by a4091e4 plus da6d5fc, with regression coverage in tests/agent/test_credential_pool_oauth_writethrough.py.
• fix(auth): inherit global Codex credential pool in profiles #34141 best fix — (+35/-9) — n/a: Replaced active-store-only Codex pool lookup with read_credential_pool("openai-codex") and added the exact named-profile regression; the keep_open review found no correctness issue, so retain as the focused Profile Codex auth can ignore global credential pool when local state is stale #34143 salvage path.
• fix(security): prevent Hermes Codex OAuth refresh-token reuse #34820 [closed] partial — (+741/-297) — n/a: Proposed broad Codex ownership, canonical-store, refresh-lock, and documentation hardening; closed after the narrow refresh serialization and User-Agent pieces were shipped in fix(auth): serialize Codex OAuth pool refresh under the auth-store lock #56233.
• fix(auth): fall back from empty codex profile state #38273 partial — (+98/-7) — n/a: Let empty profile singleton state fall back globally and made pool lookup profile-aware; its keep_open review requires source-aware refresh writeback and a forced-refresh regression before salvage.
• fix(auth): honor global active_provider in is_provider_explicitly_configured for profiles #39222 related — (+72/-3) — n/a: Restored global active_provider inheritance in profile configuration checks; the contributor keep_open review says this conflicts with the deliberate profile-scoped contract restored by 5300727.
• fix: share Codex credential pool across profiles #39352 [closed] partial — (+199/-19) — n/a: Made the global Codex pool authoritative for profile reads and writes; closed after conflicts and local-only deployment, but remains part of the shared-pool design cluster.
• fix(auth): recover codex tokens from sibling profile stores (#6653) #39796 best fix — (+1096/-27) — n/a: Adds sibling recovery for copied device_code lineage; despite the keep_open review's original arbitrary-account concern, the visible diff now requires matching pool id/source and a strictly newer last_refresh, while preserving unrelated accounts.
• fix: heal profile Codex credential cooldowns from global pool #41704 partial — (+266/-3) — n/a: Heals exhausted profile rows from same-id healthy global rows; keep-open review requires avoiding singleton overwrite of independent manual accounts and validating the production selection path.
• fix(auth): resync copied Codex profile tokens #41705 fixes — (+382/-41) — n/a: Combines global-pool fallback with sibling-profile resync by id/source, but the keep-open review rejects auth-file mtime as freshness evidence and requests an authoritative source or explicit design decision.
• fix(auth): preserve independent Codex OAuth profiles #42434 [closed] related — (+379/-95) — n/a: Protected independent labelled Codex accounts from cross-entry rewrites; closed as implemented by 761b744 and c78b3e1 with distinct-account tests on main.
• fix: share Codex auth refresh across profiles #43415 [closed] partial — (+374/-75) — n/a: Added source-aware shared-root borrowing and writeback but bundled an unrelated funciones.txt artifact; closed as a duplicate of fix(auth): share Codex OAuth across profiles #28277 after contributor objections.
• fix(auth): self-heal Codex refresh_token rotation by reimporting from ~/.codex #45261 [closed] best fix — (+182/-5) — n/a: Added relogin-required self-heal from ~/.codex/auth.json while preserving transient failures; closed after bd66e7e landed the verified implementation.
• feat(auth): shared Codex OAuth across profiles (Codex counterpart to xAI sharing) #47221 [closed] partial — (+230/-20) — n/a: Added an opt-in shared Codex auth file and locked refresh resolver; closed as superseded by the global-root write-through direction implemented by fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760.
• feat(auth): share openai-codex OAuth token across profiles via shared store #47233 [closed] partial — (+398/-6) — n/a: Added a second shared Codex store with secure write/recovery semantics; closed because fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760 implemented the same inherited-grant objective through existing global-root ownership.
• fix(auth): share Codex and xAI OAuth state across profiles #48364 [closed] partial — (+1197/-95) — n/a: Proposed shared stores for both Codex and xAI plus unrelated compression-notice changes; closed as an over-broad competitor to the narrower Codex profile fixes.
• fix(codex): write rotated OAuth grant through to global root (multi-profile rotation race) #48416 [closed] fixes — (+311/-0) — n/a: Added Codex global-root write-through mirroring xAI, with an approving review, but closed as moot for independent pools; fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760 later implemented the fallback-derived pool case on main.
• fix(auth): openai-codex multi-profile rotation write-through (#48415) #49032 [closed] fixes — (+350/-0) — n/a: Implemented Codex write-through but bundled unrelated Kanban behavior; closed after a blocking contributor review requested a focused auth-only branch, superseded by fix(auth): openai-codex multi-profile rotation write-through (#48415) #49127.
• fix(auth): openai-codex multi-profile rotation write-through (#48415) #49127 [closed] fixes — (+289/-0) — n/a: Clean auth-only resubmission of Codex singleton/pool write-through with contributor approval; closed as superseded when fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760 landed the pool-path fix.
• fix(auth): let Codex singleton reader use credential pool fallback #49170 partial — (+67/-4) — n/a: Adds direct-reader fallback to a raw pool token pair; keep-open review found that it bypasses canonical dead/cooldown selection and lacks the tests claimed in the PR body.
• fix(pool): serialize openai-codex oauth refresh #49988 [closed] partial — (+1363/-61) — n/a: Proposed path-scoped locks, CAS persistence, and exact-entry refresh isolation for singleton and manual pool credentials; closed by the author as opened by mistake.
• feat(auth): Codex OAuth write-through to global root (multi-profile parity with xAI #43589) #50632 [closed] partial — (+315/-0) — n/a: Added singleton write-through only; closed after a blocking contributor review showed the credential-pool refresh path remained stale, which fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760 subsequently fixed.
• fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #51753 [closed] fixes — (+274/-0) — n/a: Added credential-pool write-through for root-derived Codex/xAI/Nous grants with approved focused tests; closed as superseded by merged fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760.
• fix(auth): write rotated Codex/xAI pool grant through to global root (#48415) #52760 [merged] best fix — (+274/-0) — n/a: Writes rotated root-derived pool grants back to global root while preserving profile-owned grants; merged reference implementation for [Bug]: openai-codex multi-profile rotation race — rotated grant not written through to global root (Codex analog of #43589) #48415 at merge commit a4091e4.
• fix(auth): serialize Codex OAuth pool refresh under the auth-store lock #56233 [merged] partial — (+108/-1) — n/a: Serialized the Codex pool sync→POST→writeback transaction under the auth-store lock and added the Hermes User-Agent; merged reference for the concurrent refresh-token spend race.
• fix(auth): canonicalize inherited Codex refresh state #64572 best fix — (+850/-88) — n/a: Makes inherited singleton and pool state source-aware, locks and writes the canonical root directly, reloads rotated rows, and three-way-merges stale snapshots; its keep_open/high-salvage review supports this direction, but the branch is now conflicting and needs a rebase.
• fix(credential-pool): stop lost-update cooldown erasure and wrong-key quarantine #65844 [closed] related — (+276/-1) — n/a: Prevented stale snapshots from erasing cooldowns and unmatched hints from quarantining innocent keys; closed after contributor-confirmed salvage into merged fix(credential-pool): stop lost-update cooldown erasure and unmatched-hint wrong-key quarantine (#65844 salvage) #70147.
• fix(credential-pool): stop lost-update cooldown erasure and unmatched-hint wrong-key quarantine (#65844 salvage) #70147 [merged] related — (+281/-1) — n/a: Salvaged fix(credential-pool): stop lost-update cooldown erasure and wrong-key quarantine #65844's cooldown-recency merge and unmatched-hint guard with isolated regressions; merged as the reference implementation for that adjacent pool-concurrency class.

Duplicates

#15173 and #17480 share the /usage/pool fallback path; #28277, #34121, #39352, #43415, #47221, #47233, and #48364 are shared/canonical Codex-store variants; #48416, #49032, #49127, #50632, #51753, and #52760 are the root write-through chain; #39796 and #41705 are sibling-copy recovery variants. #11512 is contained in #15104, #65844 is contained in #70147, and #34820's narrow refresh-lock portion is contained in #56233.

Suggested consolidation

Author action on #64572: rebase onto current main and rerun the focused singleton/profile/pool ownership suites; this follows its recorded #6653 best-fix status and keep_open/high-salvage review without recommending a merge. Keep #34141 open as the focused #34143 salvage and keep #39796 only pending exact-head comparison with rebased #64572; close #28277, #38273, #41704, #41705, and #49170 as superseded by that canonical-source cut if their distinct reviewed concerns are covered, and close #39222 because its contributor review documents the opposite profile-boundary decision.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I6653(["issue #6653 (open)"])
    I34143(["issue #34143 (open)"])
    P64572["PR #64572 (open)"]
    P64572 -->|best fix| I6653
    P64572 -->|fixes| I34143
    class I6653 open
    class I34143 open
    class P64572 open
    class P64572 best
    class P64572 target
    click I6653 "https://github.com/NousResearch/hermes-agent/issues/6653"
    click I34143 "https://github.com/NousResearch/hermes-agent/issues/34143"
    click P64572 "https://github.com/NousResearch/hermes-agent/pull/64572"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 37 pull requests and 6 issues in this complex. Each diff was read against this issue; Assessment working set: 749 kB of PR diffs, 106 kB of issue/PR text, 51 kB of discussion (65 comments), 52 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.