feat(agent): add live upgrade — checkpoint subagents before restart and adopt orphans on startup (#71023)

[webtecnica]

What

Implements live upgrade support for Hermes Agent: when the process restarts (e.g. via /update), running subagents are checkpointed with full state before shutdown and automatically recreated on startup, preserving their progress.

Part 1 — Rich checkpoint before restart (version 2)

checkpoint_active_subagents() in delegate_tool.py now serialises the full state of running subagents to $HERMES_HOME/state/subagent_checkpoints.json. Each record contains:

• Metadata: subagent_id, parent_id, depth, goal, model, started_at, tool_count, status, last_tool
• saved_messages: The full conversation history (tool calls + results) extracted from the child agent's _session_messages and safely serialized (non-serializable fields stripped, long content truncated)
• resolved_context: The subagent's context from _subagent_goal

Checkpoint format is now version 2 (detectable by the "version": 2 field). Version 1 checkpoints (metadata only) from older code are still accepted gracefully.

Part 2 — Orphan adoption with progress preservation

adopt_orphaned_subagents() consumes the checkpoint on startup and stores the orphan data in a module-level variable _pending_orphans. Instead of just logging lost work and deleting the checkpoint:

• Full orphan records (with saved messages if available) are kept in memory
• The checkpoint warning now reports how many subagents have saved conversation state
• get_pending_orphans() provides access to the stored data

Part 3 — Auto-recreation with progress continuity

New recreate_pending_subagents(parent_agent) function:

• Reads the orphan data stored by adopt_orphaned_subagents()
• For each orphan, calls delegate_task with the saved goal and context
• When saved_messages are available (version 2+ checkpoint), formats the previous conversation as enriched context so the new subagent knows what was already done and picks up where it left off
• Includes a CONTINUATION note in the context telling the new subagent not to re-do completed work
• Called automatically from cli.py's first-turn agent initialization

Hooks

• cli.py _run_cleanup() — calls checkpoint_active_subagents() before resource teardown
• cli.py run() — calls adopt_orphaned_subagents() after the welcome banner
• cli.py first-turn init — calls recreate_pending_subagents() when the agent is first initialized, auto-re-delegating orphans

Changes separated

The unrelated "vision capability guard" change has been split into a separate PR: #26 (fix/vision-capability-guard-71027)

Testing

24 tests in tests/tools/test_delegate_checkpoint.py covering:

• _safe_serialize_messages edge cases (bytes, callbacks, truncation)
• Path resolution under $HERMES_HOME
• Version 2 checkpoint write with full state (messages + context)
• Checkpoint write without agent object (graceful fallback)
• Orphan adoption reads, stores pending, and removes checkpoint
• Corrupt checkpoint is handled gracefully
• get_pending_orphans() returns correct data
• recreate_pending_subagents() re-delegates with goal
• recreate_pending_subagents() builds enriched context from messages
• Empty/missing goals skip gracefully
• _remove_checkpoint_safe never raises

Closes

Closes #71023

Breaking Changes

None. The adopt_orphaned_subagents() return signature (int) is preserved. New functions (get_pending_orphans(), recreate_pending_subagents()) are additive.

[alt-glitch]

This was generated by AI during triage.

Related: #71027 combines the live-upgrade checkpoint proposal for #71023 with the same vision capability guard as #71024. The process-checkpoint work in #69918 is adjacent but covers detached processes rather than in-process subagents; please choose the intended scope and split/consolidate as appropriate.

[webtecnica]

Rebased onto latest main, split the unrelated vision guard into #72917, and upgraded the checkpoint to preserve full subagent state (goal, context, conversation history) with auto-recreation via recreate_pending_subagents() on startup. Ready for re-review 🚀

[teknium1]

Thanks for splitting the vision change and extending the checkpoint data. The restart-recovery gap is real on current main, but this implementation needs lifecycle and delivery rework before it is safe.

Problems

• tools/delegate_tool.py:4244 calls delegate_task without background=True. Its default is synchronous (tools/delegate_tool.py:2823, 3390-3391), unlike the normal top-level dispatch path (run_agent.py:6900-6919), so the first recovered user turn waits for the recreated work.
• cli.py:1180-1187 checkpoints every _run_cleanup; normal interactive exit reaches cleanup at cli.py:17318. Startup then adopts (cli.py:13994-14000) and first chat recreates (cli.py:12778-12790) against whichever new session starts next. This can replay work after a normal exit into an unrelated session.
• The checkpoint is removed before recreation (tools/delegate_tool.py:4134-4138), while failures are caught after pending state is cleared (4172-4175, 4259-4266), so failed recovery is not retryable.

Suggested changes

• Gate recovery on a verified update handoff plus originating-session ownership.
• Use the existing asynchronous delegation delivery path and add a real lifecycle test for non-blocking recovery and failed-dispatch retention.

Automated hermes-sweeper review.

[teknium1]

_run_cleanup() also runs for ordinary CLI exit (cli.py:17318 on current main), not only /update. With startup adoption and first-chat recreation below, this checkpoints and replays work into the next unrelated CLI session. Please gate this on a verified update/restart handoff and preserve originating-session ownership.

[teknium1]

This deletes the only durable recovery record before any recreation attempt. recreate_pending_subagents() clears pending state before dispatching and catches failures, so a failed dispatch cannot be retried after another restart. Retain or atomically update the record until each dispatch is accepted.

[teknium1]

This omits background=True. Current delegate_task resolves an omitted background value to false and waits synchronously, whereas normal top-level model delegation explicitly dispatches in the background. Recovery will block the first user turn; route it through the existing async ownership/delivery path.

[teknium1]

The saved transcript becomes context, and _build_child_system_prompt() inserts context verbatim into the new child's system prompt. That promotes previous tool output and assistant text to privileged instructions. Use a bounded, structured continuation representation that preserves the original trust boundary.

[GottZ]

This was generated by AI during triage.

Summary

One PR addresses Issue #71023. #71027 adds pre-restart checkpointing and startup re-delegation with saved goals, context, and conversation history, directly targeting lost subagent work, but the diff does not preserve live execution or safely recover results across restart.

Related pull requests

• feat(agent): add live upgrade — checkpoint subagents before restart and adopt orphans on startup (#71023) #71027 best fix — (+919/-0) — best available fix (recorded best_fix), partial implementation: adds versioned subagent checkpoints, orphan adoption, and automatic recreation with prior conversation context, addressing the reported loss of in-progress work. The contributor review marked keep_open and identifies blocking lifecycle and recovery defects: recreation omits background=True, ordinary CLI exits create cross-session replay, the checkpoint is deleted before successful dispatch, and saved transcript content crosses into privileged prompt context.

Suggested consolidation

keep open with a salvage path: retain #71027 as the best available direction, but do not merge it until the blocking contributor review is addressed. Gate checkpointing and replay on a verified update/restart handoff with originating-session ownership, dispatch recovered work asynchronously through the existing delivery path, retain or atomically advance durable recovery state until dispatch succeeds, and represent saved progress with a bounded structured continuation that preserves the original trust boundary.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I71023(["issue #71023 (open)"])
    P71027["PR #71027 (open)"]
    P71027 -->|best fix| I71023
    class I71023 open
    class P71027 open
    class P71027 best
    class P71027 target
    click I71023 "https://github.com/NousResearch/hermes-agent/issues/71023"
    click P71027 "https://github.com/NousResearch/hermes-agent/pull/71027"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 37 kB of PR diffs, 6 kB of issue/PR text, 3 kB of discussion (7 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.