Skip to content

fix(auth): apply newer hosted bootstrap session - #64612

Merged
benbarclay merged 2 commits into
mainfrom
fix/rebootstrap-newer-bootstrap-session
Jul 15, 2026
Merged

fix(auth): apply newer hosted bootstrap session#64612
benbarclay merged 2 commits into
mainfrom
fix/rebootstrap-newer-bootstrap-session

Conversation

@benbarclay

@benbarclay benbarclay commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Resolves #64610.

Production evidence

Time (UTC) Event
19:50:51 NAS minted a replacement hosted-agent bootstrap session and revoked the previous session.
19:51:41 Container boot logged [rebootstrap] no-op (not_terminal) because the local old tokens still looked healthy.
19:51:53 The old refresh session was rejected with invalid_grant: Refresh session has been revoked.
19:51:54 Hermes quarantined the OAuth state, causing NAS to start another self-heal cycle.

The affected hosted instance recorded 74 uptime resets in the sampled four-day window.

Root cause

NAS issues the replacement before restart and revokes the prior session. The boot helper only accepted HERMES_AUTH_JSON_REBOOTSTRAP after the local entry had already been quarantined. A still-present access/refresh token therefore made the helper reject the new seed, even though NAS had already revoked that local session.

Changes

  • Apply an orchestrator rebootstrap seed when its obtained_at is strictly newer than the local session.
  • Preserve the existing terminal-quarantine recovery path.
  • Refuse older, equal, malformed, or incomparable timestamps, preventing a retained env value from rolling healthy rotated credentials backward.
  • Add a distinct reseeded_newer breadcrumb for production verification.
  • Update the stage2 contract documentation and regression coverage.

Verification

  • uv run --quiet --with pytest python -m pytest tests/tools/test_docker_rebootstrap_nous_session.py tests/hermes_cli/test_nous_session_validity.py tests/hermes_cli/test_auth_nous_provider.py -q100 passed
  • uv run --quiet --with ruff==0.15.10 ruff check scripts/docker_rebootstrap_nous_session.py tests/tools/test_docker_rebootstrap_nous_session.py — clean
  • bash -n docker/stage2-hook.sh — clean
  • python -m compileall -q scripts/docker_rebootstrap_nous_session.py tests/tools/test_docker_rebootstrap_nous_session.py — clean
  • git diff --check — clean

Expected production impact

After NAS pushes a newly minted hosted bootstrap session, the first boot applies it instead of retaining the revoked predecessor. The follow-up refresh should succeed and the self-heal restart chain should stop after one cycle.

Trace-point suggestions

  • Include local and seed obtained_at plus the rebootstrap decision (terminal, newer, stale, incomparable) in the boot breadcrumb.
  • Include the affected profile/home path in nous_session_valid and NAS remint telemetry so profile-scoped auth failures are attributable without SSH inspection.

Not in this PR

  • No change to NAS lifecycle ordering or its Redis loop guard.
  • No change to profile-specific auth recovery; this fixes the verified root-session revocation loop without widening credential writes across profile boundaries.

@alt-glitch alt-glitch added type/bug Something isn't working area/auth Authentication, OAuth, credential pools area/docker Docker image, Compose, packaging provider/nous Nous Research API (OAuth) P2 Medium — degraded but workaround exists sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Jul 14, 2026
@benbarclay

Copy link
Copy Markdown
Collaborator Author

Independent review caught a blocking seed-validation gap. Follow-up commit fb74a75 now requires client_id=hermes-cli-vps plus non-empty access and refresh tokens before replacing healthy credentials; timezone-less, malformed, extreme, older, and equal timestamps remain incomparable/no-op. Fresh verification: 100 relevant tests passed; Ruff, shell syntax, compileall, and diff checks passed.

@benbarclay
benbarclay merged commit 3f2a389 into main Jul 15, 2026
32 checks passed
@benbarclay
benbarclay deleted the fix/rebootstrap-newer-bootstrap-session branch July 15, 2026 22:52
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
* fix(auth): apply newer hosted bootstrap session

* fix(auth): validate rebootstrap replacement seeds
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
* fix(auth): apply newer hosted bootstrap session

* fix(auth): validate rebootstrap replacement seeds
33hodl pushed a commit to 33hodl/hermes-agent that referenced this pull request Aug 12, 2026
* fix(auth): apply newer hosted bootstrap session

* fix(auth): validate rebootstrap replacement seeds
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
* fix(auth): apply newer hosted bootstrap session

* fix(auth): validate rebootstrap replacement seeds
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
* fix(auth): apply newer hosted bootstrap session

* fix(auth): validate rebootstrap replacement seeds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/auth Authentication, OAuth, credential pools area/docker Docker image, Compose, packaging P2 Medium — degraded but workaround exists provider/nous Nous Research API (OAuth) sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hosted-agent rebootstrap ignores newer session while local tokens still look healthy

2 participants