feat(delegation): background delegate_task restart survival (persist + auto-redispatch)

[Kyzcreig]

Summary

Background delegate_task jobs now SURVIVE gateway restarts — persist at dispatch, boot-scan for orphans, auto-redispatch with a continue-from-partial-artifact note.

Problem (two live incidents, 2026-07-10)

Background delegates run on a daemon ThreadPoolExecutor inside the gateway process. A gateway restart SIGTERMs them silently mid-flight with no trace and no resume: a 35-min research delegate (72 messages of work) died to a deploy restart; its redispatch died to a SECOND restart the same day. Kanban workers (separate processes) survived both.

Design (Momus-reviewed spec, APPROVE-WITH-CHANGES, all 4 RCs folded)

• Durable per-profile registry (tools/async_delegation_store.py): the existing self-contained task-source block persisted at dispatch; status transitions on the existing completion rail.
• Boot-scan: records still running whose boot_id != current (psutil create_time contract, same as the F2 breadcrumb) were killed by a restart → auto-redispatch with a prepended "interrupted at ; a partial artifact may exist — read it and CONTINUE" note; parent session notified via the existing completion-queue rail (new async_delegation_restarted event type).
• Redispatch cap 2 per delegation (generation-scoped attempt ids), then an honest FAILED completion — no infinite restart loops (F2 lessons).
• Config: delegation.resume_on_restart (default true) in config.yaml + DEFAULT_CONFIG (no new env vars). Rollback = flip false → exactly today's in-memory behavior.

Momus RC folds (each with a named test)

• RC-1: claimed-but-never-submitted attempts (boot dies while queued pre-submit) are reconciled on next boot and do NOT count an attempt — test_rc1_claimed_but_never_submitted_does_not_count_attempt.
• RC-2: pending restart events replay across boot-death identically to terminal events; superseded (older-generation) restart events transition to dropped(superseded), never stuck pending — test_rc2_pending_restart_event_replays_and_superseded_event_drops.
• RC-3: recoverable clean-shutdown persistence is best-effort with a bounded lock-acquire timeout; the running+dead-boot path is the guaranteed fallback (documented + enforced).
• RC-4: all file:line anchors re-grounded against current fork/main; extends the existing completion-queue crash-recovery checkpoint (no parallel mechanism).
• Momus residual: observability counter when the record cap forces synchronous-fallback degradation.

Tests

41 new tests green across 3 files (persistence store, boot-scan recovery, gateway integration); barrier-based race tests (no sleep-ordering); 506/506 broad focused sweep by the worker. Cache/alternation safe: completions re-enter as fresh turns on the existing rail.

[greptile-apps]

Greptile Summary

This PR adds restart-survival for background delegate_task jobs: a durable per-profile JSON registry (async_delegation_store.py) persists each delegation at dispatch, a boot-scan on startup claims orphaned records (owned by a now-dead PID) and re-dispatches them with a continuation prompt, and a bounded outbox pattern replays terminal/restart events across boots. All three previously-flagged P1s are resolved: the _finalize shutdown-window early-return now lets completed results flow through, temporary injection failures are capped at 900 retries (~30 min), and interrupt_for_session uses a scoped (session_key, parent_session_id) pair to avoid cancelling prior /new-reset sessions' work.

• Durable store (tools/async_delegation_store.py): file-locked JSON registry with SHA-256 record integrity, generation-scoped attempt IDs, a 2-attempt redispatch cap, and claim_recoveries implementing RC-1 (claimed-but-never-submitted boots do not consume retry budget).
• Recovery orchestration (gateway/run.py + tools/async_delegation.py): _recover_async_delegations_once iterates all served profiles under the correct credential scope, rebuilds runners via build_recovered_delegation_runner, and drains the pending outbox before the watcher starts.
• Fallback safety (tools/delegate_tool.py): if _build_durable_background_spec raises (missing boot ID, TUI session, registry cap), the delegation falls back to synchronous execution with an explanatory note rather than silently launching untracked work.

Confidence Score: 4/5

Safe to merge; the core restart-survival contract is correct and all three previously-flagged bugs are resolved. One narrow gap remains in _finalize where a runner returning "success" instead of "completed" during the shutdown window has its result silently discarded and the job re-dispatched.

The shutdown-window guard in _finalize uses status != "completed" while append_terminal explicitly treats both "completed" and "success" as terminal-success. For current gateway delegate_task runners this gap is inert (they always emit "completed"), but ACP-mode external runners could return "success", causing completed work to be re-executed on the next boot.

tools/async_delegation.py lines 361 and 640 (_finalize and _finalize_batch shutdown-window guards)

Important Files Changed

Filename	Overview
tools/async_delegation_store.py	New 886-line durable registry: file-locked JSON store with SHA-256 integrity, outbox/acknowledgement pattern, claim_recoveries boot-scan, and cancel_matching. Minor unguarded record["attempt"] access in transition_owned.
tools/async_delegation.py	Wires durable store into dispatch/finalize paths and adds recover_async_delegations + enqueue_pending_outbox. The shutdown-window guard (status != "completed") fixes the previous P1 but omits "success", leaving a gap for ACP runners.
gateway/run.py	Adds boot-scan recovery hook, profile-scoped credential re-resolution, bounded retry cap (900 cycles), profile-matching guard for session source lookup, and scoped interrupt_for_session call. All previously flagged P1s addressed.
tools/delegate_tool.py	Adds _build_durable_background_spec, build_recovered_delegation_runner, and _recovery_spec plumbing. Credentials absent from persisted spec and re-resolved on recovery. Sync fallback on registry rejection is correct.
gateway/status.py	Adds is_boot_id_alive using psutil create_time contract; compact and correct.
tools/process_registry.py	Adds async_delegation_restarted formatter; low-risk additive change.
gateway/slash_commands.py	Adds scoped interrupt_for_session on /stop; mirrors the main interrupt path correctly.
hermes_cli/config.py	Adds delegation.resume_on_restart default (true) to DEFAULT_CONFIG; clean single-line addition.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GW as Gateway Boot
    participant Store as async_delegation_store
    participant Exec as ThreadPoolExecutor
    participant Queue as completion_queue

    Note over GW: Normal dispatch
    GW->>Store: persist_dispatch(delegation_id, durable_spec, boot_id)
    Store-->>GW: record with attempt_id
    GW->>Exec: submit _worker
    Exec->>Store: mark_submitted(attempt_id)
    Exec->>Exec: runner()
    Exec->>Store: append_terminal(attempt_id, result, status)
    Store-->>Queue: terminal event queued
    Queue-->>GW: watcher delivers completion notification

    Note over GW: Gateway restart kills workers mid-flight
    Note over GW: Next boot recovery path
    GW->>Store: claim_recoveries(current_boot_id)
    Store->>Store: find running records with dead boot_id
    Store->>Store: increment generation, write restart outbox event
    Store-->>GW: claimed records
    GW->>Exec: submit recovery _worker with continuation note
    GW->>Store: enqueue_pending_outbox(current_boot_id)
    Store-->>Queue: restart and pending terminal events
    Queue-->>GW: watcher delivers restarted notification

    Note over GW: Shutdown with recoverable=True
    GW->>Store: transition_owned(ids, recoverable) best-effort
    GW->>Exec: interrupt signals sent to workers
    Note over Store: Record stays recoverable, next boot re-claims it

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GW as Gateway Boot
    participant Store as async_delegation_store
    participant Exec as ThreadPoolExecutor
    participant Queue as completion_queue

    Note over GW: Normal dispatch
    GW->>Store: persist_dispatch(delegation_id, durable_spec, boot_id)
    Store-->>GW: record with attempt_id
    GW->>Exec: submit _worker
    Exec->>Store: mark_submitted(attempt_id)
    Exec->>Exec: runner()
    Exec->>Store: append_terminal(attempt_id, result, status)
    Store-->>Queue: terminal event queued
    Queue-->>GW: watcher delivers completion notification

    Note over GW: Gateway restart kills workers mid-flight
    Note over GW: Next boot recovery path
    GW->>Store: claim_recoveries(current_boot_id)
    Store->>Store: find running records with dead boot_id
    Store->>Store: increment generation, write restart outbox event
    Store-->>GW: claimed records
    GW->>Exec: submit recovery _worker with continuation note
    GW->>Store: enqueue_pending_outbox(current_boot_id)
    Store-->>Queue: restart and pending terminal events
    Queue-->>GW: watcher delivers restarted notification

    Note over GW: Shutdown with recoverable=True
    GW->>Store: transition_owned(ids, recoverable) best-effort
    GW->>Exec: interrupt signals sent to workers
    Note over Store: Record stays recoverable, next boot re-claims it

Loading

_{Reviews (4): Last reviewed commit: "fix(delegation): restore either-selector..." | Re-trigger Greptile}