Skip to content

fix(cli): honor --resume/--continue in oneshot (-z) — rebase + real-SQLite coverage - #80799

Open
blairjordan wants to merge 5 commits into
NousResearch:mainfrom
blairjordan:fix/oneshot-resume-hydration-completion
Open

fix(cli): honor --resume/--continue in oneshot (-z) — rebase + real-SQLite coverage#80799
blairjordan wants to merge 5 commits into
NousResearch:mainfrom
blairjordan:fix/oneshot-resume-hydration-completion

Conversation

@blairjordan

@blairjordan blairjordan commented Aug 7, 2026

Copy link
Copy Markdown

What

Carries forward #57859 (fix/oneshot-resume-hydration, @nathansmithopenclaw-alt): hermes --resume <id> -z "<prompt>" (and --continue [name]) chains — the session's prior transcript loads as conversation history and the turn appends to the SAME session id. Ids that don't exist yet are created on first use, so scripted callers can mint a stable session key per conversation up front and pass it on every turn.

This closes #49195, and addresses everything from @teknium1's review of #57859:

  1. Rebased onto current main. fix(cli): honor --resume/--continue in oneshot (-z) — hydrate and chain the session #57859 predates the MCP-discovery-wait (ensure_mcp_discovery_before_agent_build) and requested_provider/usage_file additions to _run_agent/run_oneshot — this carries the resume hydration forward alongside all three.
  2. Real-SQLite integration coverage, not mock-only: two/three-turn _run_agent() calls against a real, temp-path-backed SessionDB (only the LLM call is faked — the stand-in drives the genuine AIAgent._flush_messages_to_session_db against real disk), then a reload through a brand-new SessionDB instance — the shape a second hermes -z process invocation would see. The row-count/dedup assertion is verified non-vacuous by mutation (breaking the real identity-dedup makes it fail).

Update: this PR went through two independent adversarial reviews after opening (see below) that found and fixed three additional real issues beyond what's described above. Net diff vs. the originally-opened version: a hardened exit path for -z --resume/--continue failures, a path-traversal guard on caller-minted ids, and ID-or-title resolution for --resume matching its own documented contract.

Why this needs to exist

hermes -z (oneshot) is the interface a piped/scripted caller uses precisely to avoid chat's banner/box/session chrome on stdout. Every such caller that wants multi-turn continuity — a gateway running a chat seat over -z one-shots, a cron worker, a CI agent pipeline driving Hatchet-style multi-step workflows — currently gets an amnesiac agent: --resume/--continue are accepted (exit 0, no warning) and silently dropped, so every turn starts from zero. This is the third independent fix attempt at this exact gap (#40333 closed in error unrelated to merit, #49204 takes the inverse "reject the flag" minimal path) — the bug report, the review signal, and the number of independent contributors who've hit this all say it's a real, wanted capability.

Relationship to existing PRs/issues

Change

  • hermes_cli/oneshot.py:
    • run_oneshot(resume=...)_run_agent(resume=...)_load_resume_history(session_db, resume): resolves the compression chain via resolve_resume_session_id, loads messages via get_messages_as_conversation (dropping session_meta rows), best-effort reopen_session, and pins AIAgent(session_id=...). A broken session store degrades to a stateless turn — -z never gets less reliable than before.
    • The resume load happens inside the try/finally that owns session_db, matching that block's own stated invariant that the connection is always closed (not reachable as a live bug today — _load_resume_history's own internals are individually guarded — but it contradicted the comment directly above it).
  • hermes_cli/main.py:
    • Both oneshot dispatch sites (Termux fast-path and main) resolve resume=_resolve_oneshot_resume(args) and forward it through _run_and_exit_oneshot.
    • --resume resolves by ID or title, exactly like cmd_chat and the flag's own _parser.py help text ("Resume a previous session by ID or title"). Without this, --resume "my project" would silently mint an empty, wrongly-named session instead of resuming — the identical "documented behavior silently doesn't happen in -z" bug class as hermes -z (oneshot) silently ignores --resume / --continue (session never hydrated) #49195 itself, and the exact gap sibling PR fix: preserve one-shot resume context #70136 was reviewed down for. Resolution still falls through to create-on-first-use when nothing matches.
    • Path-traversal guard on caller-minted ids (_is_unsafe_new_session_id): a create-on-first-use id becomes a filename downstream (SessionDB's session-file removal path is unsanitized), so hermes --resume ../../x -z ... would mint a row that a later sessions delete/prune unlinks outside the sessions directory (CWE-22). Mirrors the gateway's own _is_path_unsafe guard at its equivalent entry boundary. Only applied to values about to become new ids — a title containing / that resolves to a real session is untouched.
    • Hardened exit path (_fail_and_exit_oneshot): all three _resolve_oneshot_resume failure branches (unmatched --continue by name, no prior session for bare --continue, path-unsafe --resume id) now exit via _cleanup_oneshot_runtime() + _exit_after_oneshot() instead of a bare sys.exit(2). By the time oneshot dispatch runs, _prepare_agent_startup() has already discovered plugins and can have started MCP server subprocesses — unwinding through normal interpreter finalization both orphans them and re-enters the native finalizer teardown _exit_after_oneshot exists to skip (Bug: hermes -z prints successful response then aborts during shutdown with exit 134 #30387/Bug: hermes -z aborts with SIGABRT during teardown on AL2023 #43055).
    • --continue resolves by name/most-recent exactly like interactive chat; an unresolvable --continue errors (nothing sensible to chain onto).
  • hermes_cli/_parser.py-z help documents ID-or-title + create-on-first-use + the --continue error semantics.
  • With no --resume/--continue, behavior is byte-for-byte unchanged.

One open question for a maintainer, not resolved here

Whether --resume <title> should resolve at all in oneshot was reviewed both ways during this PR's own review cycle: one review argued for matching the documented ID-or-title contract (implemented, above); a second flagged a theoretical hijack risk — a caller-minted key that happens to collide with an existing session's human-readable title would land on the wrong session. I kept the resolution because it's the same resolution function and ordering (exact id first, title fallback) that interactive chat already uses today, so this isn't a new risk class introduced by this PR, and it matches what --parser.py promises and what #70136's review already asked for. Flagging explicitly in case a maintainer weighs this differently.

Tests

  • tests/hermes_cli/test_oneshot_resume.py: 26 passed (7 original mock-based + 2 real-SQLite integration + 3 new integration cases — third-turn accumulation, compression-chain redirect, unknown-id create-on-first-use — + 14 in a new TestResolveOneshotResume covering title resolution, path-traversal rejection (5 shapes), the hardened-exit contract, and --continue resolution).
  • tests/hermes_cli/ -k "resume or continue or oneshot or session": 203 passed, 5 failed — all 5 are pre-existing dashboard-auth 401s unrelated to this change (confirmed against a clean-main baseline in the same sandbox).
  • Full tests/hermes_cli/ suite: no regressions vs. a clean-main baseline in the same sandbox (identical pre-existing failure count; the only pass-count delta matches this PR's own new tests).
  • Manual verification, live, against a real deployed nousresearch/hermes-agent:latest container: planted a codeword on turn 1 under a caller-minted --resume id, recalled it correctly on turn 2 under the same id, confirmed a different --resume id sees no cross-session leakage.

Review history

This branch was reviewed twice after opening, independently and adversarially (one focused on correctness of the session/exit-path mechanics, one on completeness against this repo's own CONTRIBUTING.md/AGENTS.md contribution rubric and against gaps sibling PRs were reviewed down for). Both found real, non-overlapping issues; both are folded into the current diff, verified together (not just individually) before pushing. Happy to expand on either review's specific findings if useful for re-review.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

Update: two more independent reviews (security, concurrency/production-robustness)

After the round above, this branch went through two more adversarial passes — one hunting security issues specifically, one focused on concurrency and production robustness given the actual motivating use case (a Hatchet-orchestrated pipeline running multiple concurrent multi-step chains against the same session store). Both found real, distinct, confirmed issues, now folded into the diff:

Security:

  • SessionDB._remove_session_files (hermes_state.py) interpolated session_id raw into a glob pattern. An id of literally * — legal under the path-traversal guard, since * isn't a traversal character — expands request_dump_{id}_*.json into request_dump_*_*.json and deletes every other session's request dumps. --resume made this CLI-reachable for the first time (previously gateway-only). Fixed with glob.escape.
  • _is_unsafe_new_session_id only ported one of the three checks the gateway's own equivalent entry boundary applies together. Added control-character rejection (CR/LF can forge log lines; ESC can inject terminal escapes into the rejection message itself) and the same 256-char length cap the gateway enforces (past NAME_MAX the session snapshot file becomes permanently uncreatable). Rejection messages now echo the offending value through a safe-echo helper so rejecting a hostile id can't itself be the injection vector.
  • Investigated and found clean: prompt injection via replayed history (nothing re-dispatches replayed tool calls; approval state isn't restorable from transcript content), --yolo bleed across resumed turns, toolset/model scope leakage, unbounded-history growth (handled by existing compression machinery), torn reads.
  • Flagged, not fixed (product decisions, not bugs): create-on-first-use resolving by title before falling back to "use as-is" (same tradeoff as the title-resolution decision above); reopen_session clearing end_reason unconditionally (matches interactive chat's existing behavior — not a regression, just an existing shared question); no single-writer enforcement on a caller-minted id (not a vulnerability, but worth a docs line).

Concurrency/production-robustness — this is the one I'd flag as most load-bearing given the actual use case:

  • The two --resume reads had no lock-contention patience, unlike every write path in this codebase (which gets 20–60s of jittered retry). Off WAL — which is the default on any pre-3.51.3 SQLite (i.e. stock CPython 3.11/3.12/3.13) or NFS/SMB/ZFS — a sibling connection holding the write lock for even a few seconds (a large FTS append, an incremental merge, a checkpoint, VACUUM — all real, existing code paths) made the read raise database is locked, which the resume loader already swallowed to a stateless degrade. Concretely: a resumed turn would silently run with zero prior context, exit 0, and append its context-less answer into the middle of an otherwise-good transcript — full token cost paid, wrong answer produced, chain poisoned for every downstream step, with no error signal at all. Reproduced against a real SessionDB under a held write lock. Fixed by giving the read path the same lock-patience budget the write path already has, escalating to a new ResumeStoreContendedError (routed through the existing non-zero-exit error path, so an orchestrator can retry) only if the lock never actually clears — a genuinely broken store still degrades to stateless as before.
  • Everything else probed held up under real, measured load: throughput overhead from resume chaining is ~1.7x DB-side cost at N=24 concurrent chains (well under 1% of a real LLM turn's wall-clock), retries never corrupt state (the healing path already handles an abandoned tool-call tail), concurrent same-id minting degrades gracefully via the existing upsert semantics, and multi-hop (3+ turn) chaining behaves identically to the tested 2-turn case.
  • Flagged, not fixed (out of scope for this diff): a double SessionDB open per resume call (main.py resolves the id, then oneshot.py opens its own store — measurable but small, and fixing it means threading a connection through a wider call chain than this PR should touch); bare --continue under concurrency resolves to the workspace's most-recently-used session, which is correct per the flag's definition but means concurrent callers in the same repo must use --resume <stable-id>, never bare --continue — worth a docs note for anyone building an orchestrator on this.

Tests: tests/hermes_cli/test_oneshot_resume.py now 40 passed (up from 26). tests/hermes_cli/ -k "resume or continue or oneshot or session": 217 passed, same 5 pre-existing unrelated dashboard-auth failures every prior review round hit.

nathansmithopenclaw-alt and others added 2 commits August 7, 2026 14:24
hermes -z parsed --resume but silently dropped it: both dispatch sites
called run_oneshot() without the flag, and the agent was built with no
session id and no history — every one-shot turn was stateless.

Now:
- run_oneshot(resume=...) loads the session transcript as
  conversation_history (walking compression chains via
  resolve_resume_session_id, dropping session_meta rows) and pins the
  agent to the SAME session id, mirroring the interactive resume path.
- Create-on-first-use: an id Hermes hasn't seen is used as-is with no
  history, so scripted callers (gateways, cron workers) can mint stable
  session keys up front and pass them on every turn — no output parsing.
- --continue resolves by name / most-recent exactly like interactive
  chat (_resolve_oneshot_resume).
- Message flush dedup is inherited: history dicts seeded via
  conversation_history are skipped by identity, so resumed transcripts
  are never re-written to the store.
- Best-effort: a broken session store degrades to a stateless turn.

Verified live: ZEBRA planted and recalled across two -z invocations on a
caller-minted id. tests/hermes_cli/test_oneshot_resume.py (7 new);
tests/hermes_cli 7804 passed — failure set byte-identical to the
pre-change baseline (259 pre-existing, attributed by clean-tree diff).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebases nathansmithopenclaw-alt's NousResearch#57859 onto current main (which had
since landed the MCP-discovery wait and requested_provider/usage_file
params in _run_agent — this carries both forward alongside the resume
hydration) and addresses the two points from @teknium1's review:

- The existing test_oneshot_resume.py coverage is entirely mock-based
  (SessionDB, AIAgent, and _load_resume_history's own dependencies are
  all MagicMocks) — real create-on-first-use behavior, a second-process
  reload, and duplicate-row risk on the resume-and-reflush cycle were
  never actually exercised against disk.
- Adds TestOneshotResumeIntegration: two _run_agent() calls against a
  real, temp-path-backed SessionDB (only AIAgent is replaced, with a fake
  that performs the SAME real create_session/append_messages_batch calls
  a genuine turn's flush does), then a THIRD read through a brand-new
  SessionDB instance — the same shape a second `hermes -z` process
  invocation would see. Asserts the full two-turn transcript loads in
  order with exactly 4 rows, not 8: turn 1's messages are seeded as
  conversation_history and never re-appended by the flush's identity
  dedup.

tests/hermes_cli/test_oneshot_resume.py: 9 passed (7 existing + 2 new).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard area/sessions Session lifecycle, resume, persistence, history needs-decision Awaiting maintainer decision before any implementation sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 7, 2026
blairjordan and others added 3 commits August 7, 2026 15:06
Three gaps in the oneshot resume path, found reviewing NousResearch#80799 against
CONTRIBUTING.md and the review findings on sibling PR NousResearch#70136.

1. `--resume` bypassed the ID-or-title resolution contract.
   `_parser.py` documents `--resume SESSION` as "by ID or title" and
   `cmd_chat` resolves titles via `_resolve_session_by_name_or_id`.
   `_resolve_oneshot_resume` passed the value through verbatim, so
   `hermes --resume "my project" -z "..."` silently minted an empty
   session literally named "my project" instead of resuming — the same
   silently-drops-a-flag class as the bug being fixed, and the exact
   finding teknium1 gave NousResearch#70136. Now resolves first; an unresolved value
   still falls through to create-on-first-use, so the design intent
   (scripted callers minting stable ids) is preserved. The asymmetry was
   visible inside the PR itself: `--continue "name"` already resolved.

2. Create-on-first-use was an unguarded entry boundary for
   caller-supplied session ids. A session id becomes a filename
   downstream — `SessionDB._remove_session_files` builds
   `sessions_dir / f"{id}.json"` and globs `request_dump_{id}_*.json`
   without sanitizing — so `hermes --resume ../../x -z "hi"` minted a
   row that a later `sessions delete`/`prune` unlinks outside the
   sessions dir (verified: the victim file is removed). `gateway/session.py`
   already rejects exactly this at its own entry boundary
   (`_is_path_unsafe`, CWE-22); the new CLI boundary now does too.
   Path-unsafe values that *resolve* to a real session are still fine —
   the guard only applies to values about to become new ids, so titles
   containing `/` keep working.

3. The integration test asserted a dedup property it never exercised.
   `_RecordingFakeAgent` appended only its own two new messages, so
   "4 rows, not 8" held by construction regardless of whether the flush's
   identity-based seeded-history dedup worked. It now drives the real
   `AIAgent._flush_messages_to_session_db` with the real
   `messages = conversation_history + this turn` shape; break the identity
   contract and the assertion fails (verified by mutation).

Also adds the coverage teknium1's review would still have flagged:
`_resolve_oneshot_resume` had zero tests (title, exact id, unknown id,
path-unsafe, `--continue` by name / bare / unmatched), a third-turn
accumulation case, and the compression-chain redirect — a caller passing
the pre-compression parent id must land on the continuation child, with
this turn's writes going there and not into the dead parent.

Docs: `-z` help and the `hermes_cli/oneshot` module docstring now state
ID-or-title, create-on-first-use, and that an unmatched `--continue`
errors while an unmatched `--resume` does not.

tests/hermes_cli/test_oneshot_resume.py: 26 passed (was 9).
Reconciles two independent reviews of this branch (review-branch /
review-completeness) into one commit:

From the correctness review:
- _resolve_oneshot_resume's three failure paths (unresolvable --continue
  by name, no prior session for bare --continue, and the new path-unsafe
  --resume id) called sys.exit(2) directly. By that point
  _prepare_agent_startup() has already discovered plugins and can have
  started MCP server subprocesses, so unwinding through normal
  interpreter finalization both orphans them (only
  _cleanup_oneshot_runtime() reaps them) and re-enters the native
  finalizer teardown _exit_after_oneshot exists to skip (NousResearch#30387, NousResearch#43055).
  Added _fail_and_exit_oneshot() and routed all three through it.
- _load_resume_history() was called before the try that owns session_db,
  contradicting that try's own stated invariant ("always closed").
  _load_resume_history's own internals are individually guarded so this
  wasn't reachable today, but moved it inside for real.

Test fallout from the above: _exit_after_oneshot does a genuine os._exit,
not a raise, so the three tests that previously asserted
`pytest.raises(SystemExit)` around these paths would now hard-kill the
test process. Rewrote them (_assert_hardened_exit helper) to patch
_cleanup_oneshot_runtime/_exit_after_oneshot and assert on the calls,
matching the pattern the correctness review's own new test used.

Verified: tests/hermes_cli/test_oneshot_resume.py 26/26 passed.
tests/hermes_cli/ -k "resume or continue or oneshot or session":
203 passed, same 5 pre-existing dashboard-auth 401s as both individual
review branches (unrelated to this change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconciles two more independent reviews of this branch into one commit,
on top of the prior reconciliation (983b55f):

Security review:
- hermes_state.py: SessionDB._remove_session_files interpolated
  session_id RAW into a glob pattern (request_dump_{id}_*.json). An id
  of literally '*' (legal under the existing path-traversal guard - it's
  not a traversal character) expands to request_dump_*_*.json and
  unlinks every OTHER session's request dumps. Fixed with glob.escape.
  Pre-PR this was gateway-only reachable; --resume makes it CLI-reachable.
- hermes_cli/main.py: _is_unsafe_new_session_id only ported one of the
  three checks the gateway's equivalent entry boundary
  (gateway.session._is_path_unsafe) actually applies together: added
  control-character rejection (CR/LF log-line forgery, ESC terminal
  escape injection) and the same 256-char length cap (past NAME_MAX the
  session snapshot file becomes permanently uncreatable). Rejection
  messages now echo the offending value through a safe-echo helper so
  the rejection itself can't replay the injection.

Concurrency/production-robustness review:
- hermes_cli/oneshot.py: the two --resume reads (resolve_resume_session_id,
  get_messages_as_conversation) had no lock-contention patience, unlike
  every write path in this codebase (20-60s jittered retry). Off WAL --
  the default on any pre-3.51.3 SQLite (stock CPython) or NFS/SMB/ZFS --
  a sibling connection holding the write lock for a few seconds (a large
  FTS append, incremental merge, checkpoint, VACUUM) made the read raise
  "database is locked", which _load_resume_history already swallowed to
  a stateless degrade. Result: a resumed turn silently runs with NO prior
  context, exits 0, and appends its context-less answer into the middle
  of the chain -- full token cost, wrong answer, poisoned transcript for
  every downstream step. This is the exact failure mode a Hatchet-style
  concurrent multi-workflow caller (the motivating use case for this
  feature) will hit under real contention. Added _read_with_lock_patience:
  waits out transient lock contention on the same budget the write path
  already uses; raises ResumeStoreContendedError (routed through
  run_oneshot's existing non-zero-exit error path, so an orchestrator can
  retry) only when the lock never clears. A genuinely broken store still
  degrades to stateless as before -- only the contended-but-healthy case
  now waits instead of silently losing context.

Verified together: tests/hermes_cli/test_oneshot_resume.py 40/40 passed
(26 prior + 6 lock-patience + 5 control-char/glob-escape + others).
tests/hermes_cli/ -k "resume or continue or oneshot or session":
217 passed, same 5 pre-existing dashboard-auth 401s as every prior
review branch (unrelated to this change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@blairjordan blairjordan closed this Aug 7, 2026
@blairjordan blairjordan reopened this Aug 7, 2026
@blairjordan

Copy link
Copy Markdown
Author

This has been running in production for us since 2026-08-07, so I can speak to the two risk labels with real operational evidence rather than just code review.

Context: we hand-patched oneshot.py / main.py / hermes_state.py as a local override (pinned to a specific base image digest) ahead of this PR landing, because we have an internal Buildkite-triggered pipeline ("dev-loop") that chains 5–6 hermes -z --resume <id> calls into one continuous session per run — classify → investigate_bug/plan_change → implement_change → create_pr → address_review_feedback, each step resuming the session the previous step produced. It's the exact multi-turn --resume chaining this PR adds. That pipeline has since authored and merged several of its own PRs against its own repo, purely by chaining oneshot turns through the same session id.

On sweeper:risk-compatibility: worth being upfront that our first attempt at this override did break prod — we patched against an independently-resolved hermes-agent:latest pull from a dev laptop, and the box's own docker compose build --pull resolved :latest to a different digest at deploy time. The override's hermes_state.py ended up missing methods (flush_token_counts, append_messages_batch, preflight_db_writability) that run_agent.py/kanban_db.py call on every turn — every message failed to persist for a few minutes until we caught it (AttributeError: 'SessionDB' object has no attribute ... on every message) and hotfixed. Root cause was purely version drift between the patch and the actual running image, not the resume/continue logic itself. Fix was pinning the base image to an exact digest and diffing method signatures against that exact digest before redeploying — since then, zero session-related errors or SQLite lock-contention failures in production logs. If anything, this is an argument for landing the fix upstream: every downstream consumer who wants --resume in oneshot today has to hand-roll this same override and is exposed to the same drift risk we hit.

On sweeper:risk-session-state: before redeploying the corrected patch, we did a live correctness check — planted a codeword in one caller-minted --resume turn, recalled it correctly on a second turn against the same session id, and confirmed plain (non-resume) oneshot calls were unaffected. In production since, the chained dev-loop steps above have run repeatedly with no session corruption, no cross-session bleed, and no mis-associated context that we've observed.

Happy to point at specific commits/logs if it helps move this past needs-decision, or to expand test coverage further if there's a specific scenario the maintainers want covered beyond what's already in the PR.

@blairjordan

Copy link
Copy Markdown
Author

Separately — CI hasn't run on this PR yet (no checks show up), which I think is the first-time-contributor workflow-approval gate. If a maintainer can approve the workflow run, that'd help get this off the "no CI signal" pile — happy to push a rebase if that's easier to trigger a fresh run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/cli CLI entry point, hermes_cli/, setup wizard needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes -z (oneshot) silently ignores --resume / --continue (session never hydrated)

3 participants