fix(gateway): preserve-and-prompt on restart instead of silently skipping interrupted work

[Kyzcreig]

What & why

Today when the gateway restarts mid-turn (deploy, OOM, safe-restart.py, crash), the resume note tells the model to "skip any unfinished work from the conversation history" (gateway/run.py) — silently dropping an interrupted long task. The user's hour-long turn just vanishes.

This changes the behavior to preserve-and-prompt: on reconnect the agent surfaces exactly where it stopped — what it had completed, what was in-flight — and asks whether to pick it up or redirect, then waits. It does not auto-continue an arbitrary in-flight turn unattended (that path was explicitly descoped — it carried an outage-class self-restart-loop risk for near-zero marginal value when the human is present).

Design (super-passed: 4 Opus review passes BLOCK→BLOCK→AWC→APPROVE + a code-review BLOCK→fix→verify)

• (d) Surface-and-ask resume note. In the _is_resume_pending arm, when the reloaded tail is a synthetic interrupt-close turn, emit a "summarize completed + in-flight, then ask go-or-redirect" note instead of "skip unfinished work". The B14 backstop clause ("any restart/shutdown command already ran — do NOT re-execute or verify it") is retained in all branches. Note placement is unchanged (prompt cache safe); only the text differs.
• Structural discriminator (not content-equality). The synthetic close turn is flagged via finish_reason="interrupt_close" — an existing SQLite column that round-trips — so a model that merely emits the literal text "Operation interrupted." can never mis-trigger the branch. (An in-memory _interrupt_close bool is also set for the pre-persist path.)
• Boundary hardening. close_interrupted_tool_sequence now makes every interrupt shape API-valid (tool tail, partial-text tail, dangling tool_calls tail) and flagged, so the reload is valid and the summary accurate.
• INV-D7 deterministic interlock. Because the resume turn is internal (no human message), the executor deterministically blocks forward/mutating tool calls on that turn until the next genuine human message — so "never auto-continue" is enforced, not just prompted. Pure reads are allowed; a blocked forward call emits RESUME_AUTOCONTINUE_VIOLATION. Turn-scoped (holds across model rounds), cleared only on the next human turn.

Tests (17 focused, all green)

• tests/gateway/test_resume_flag_roundtrip.py — the load-bearing gate: persists a real interrupted turn to a temp HERMES_HOME via the actual SessionDB, reloads via get_messages, and asserts the discriminator survives SQLite (a plain "Operation interrupted." with finish_reason=None reloads as NOT an interrupt-close — collision-proof).
• test_resume_surface_and_ask.py — note text: surface-and-ask branch has no "skip unfinished work" / no "continue"; idle branch unchanged; collision control; new-message case.
• test_resume_interlock.py — the interlock blocks a mock model that ignores the note, is turn-scoped (round 2 still blocked), and resets on the next human turn.
• test_interrupt_close_flag.py, test_resume_summary_only_reset.py, test_resume_note_cache_safe.py.

Full-suite failures on this branch were verified pre-existing (baseline fork/main produces the same transport/credential-test failures under a hermetic sandbox; this branch is +net-passing with zero regressions attributable to the diff).

Scope: Phase 3 (cross-path restart coalescing) is intentionally a separate PR.

[github-actions]

🔎 Lint report: feat/gateway-preserve-prompt-restart vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 12263 on HEAD, 12243 on base (🆕 +20)

🆕 New issues (13):

Rule	Count
unresolved-attribute	9
invalid-assignment	3
invalid-argument-type	1

First entries

tests/agent/test_resume_interlock.py:52: [unresolved-attribute] unresolved-attribute: Unresolved attribute `tool_delay` on type `AIAgent`
tests/agent/test_resume_interlock.py:53: [unresolved-attribute] unresolved-attribute: Unresolved attribute `compression_enabled` on type `AIAgent`
tests/agent/test_resume_interlock.py:50: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_cached_system_prompt` on type `AIAgent`
tests/agent/test_resume_interlock.py:109: [unresolved-attribute] unresolved-attribute: Unresolved attribute `tool_progress_callback` on type `AIAgent`
tests/agent/test_resume_interlock.py:128: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_resume_summary_only` on type `AIAgent`
tests/gateway/test_resume_flag_roundtrip.py:65: [invalid-argument-type] invalid-argument-type: Argument to bound method `SessionDB.append_message` is incorrect: Expected `str`, found `None`
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
tests/agent/test_resume_interlock.py:54: [unresolved-attribute] unresolved-attribute: Unresolved attribute `save_trajectories` on type `AIAgent`
gateway/run.py:17963: [invalid-assignment] invalid-assignment: Object of type `Literal[True]` is not assignable to attribute `_resume_summary_only` on type `(Any & ~None) | AIAgent`
tests/agent/test_resume_interlock.py:123: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `tool_progress_callback`
tests/agent/test_resume_interlock.py:103: [unresolved-attribute] unresolved-attribute: Object of type `AIAgent` has no attribute `_resume_summary_only`
gateway/run.py:17983: [invalid-assignment] invalid-assignment: Object of type `Literal[False]` is not assignable to attribute `_resume_summary_only` on type `(Any & ~None) | AIAgent`
tests/agent/test_resume_interlock.py:51: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_use_prompt_caching` on type `AIAgent`

✅ Fixed issues (2):

Rule	Count
unresolved-attribute	2

First entries

run_agent.py:3089: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

Unchanged: 6414 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[greptile-apps]

Greptile Summary

This PR replaces the gateway's silent "skip unfinished work" resume behaviour with a preserve-and-prompt flow: on reconnect after a restart, the agent surfaces what it completed and what was in-flight, then waits for the user to decide rather than silently dropping the interrupted turn.

• Structural discriminator (finish_reason=\"interrupt_close\") is set on synthetic close turns and survives the SQLite round-trip, so the resume branch keys on a DB column rather than fragile content-equality. An in-place mutation path (update_message_finish_reason) re-persists the flag for already-flushed assistant-text tails.
• _block_resume_summary_only_tools interlock (INV-D7) enforces "no auto-continue" at the executor level for the entire resume turn, not just the first tool round; cleared only on the next genuine human message.
• Boundary hardening in close_interrupted_tool_sequence makes all three interrupt shapes (tool-tail, dangling assistant(tool_calls), partial-text tail) API-valid and flagged before persist.

Confidence Score: 4/5

Safe to merge with one targeted fix: the in-place re-persist path in run_agent.py silently skips update_message_finish_reason when append_message returns a non-integer row ID, with no log warning to surface the failure.

The in-place assistant-text-tail re-persist path silently skips update_message_finish_reason when append_message returns a non-integer row ID, with no log warning. If this ever occurs, the surface-and-ask branch invisibly reverts to the old skip-unfinished-work behaviour with no observable signal.

run_agent.py around the isinstance(_row_id, int) guard in _flush_messages_to_session_db, and hermes_state.py update_message_finish_reason for the missing rowcount check.

Important Files Changed

Filename	Overview
gateway/run.py	Adds _is_interrupt_close_tail, _build_resume_pending_message, and _clear_resume_summary_only_for_human_turn helpers; replaces the old "skip unfinished work" resume note with the new conditional surface-and-ask note; sets/clears the _resume_summary_only interlock flag across all message-handling branches.
run_agent.py	Extends _flush_messages_to_session_db to track SQLite row IDs per flushed message dict and re-persist finish_reason="interrupt_close" via update_message_finish_reason when a message is mutated in-place; the re-persist path silently skips if append_message returns a non-integer, with no log warning.
agent/message_sanitization.py	Extends close_interrupted_tool_sequence to handle three interrupt shapes (tool-tail, dangling assistant(tool_calls), partial-text assistant tail); sets both _interrupt_close: True (in-memory) and finish_reason="interrupt_close" (SQLite-durable discriminator) on all flagged closes.
agent/tool_executor.py	Adds _block_resume_summary_only_tools interlock that blocks all tool calls while _resume_summary_only is True, emitting RESUME_AUTOCONTINUE_VIOLATION only for non-read-only tools; wired into both execute_tool_calls_concurrent and execute_tool_calls_sequential as an early return.
hermes_state.py	Adds update_message_finish_reason targeted UPDATE for re-persisting the finish_reason column on an already-written row; does not verify rowcount, so a missed row update is a silent no-op.
tests/gateway/test_resume_flag_roundtrip.py	Load-bearing acceptance gate: uses real SessionDB to verify the finish_reason="interrupt_close" discriminator survives a full append-reload cycle (including the in-place mutation / re-persist path) and that a plain "Operation interrupted." with no flag does not trigger the surface-and-ask branch.
tests/agent/test_resume_interlock.py	Covers forward-tool blocking, turn-scoped persistence of the interlock across multiple rounds, read-only tool suppression of VIOLATION logs, and normal-turn pass-through; uses mock handle_function_call to assert zero real invocations.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GW as Gateway
    participant DB as SessionDB
    participant Exec as tool_executor
    participant Model as LLM

    Note over GW,Model: Mid-turn gateway restart
    GW->>DB: "append_message finish_reason=interrupt_close"
    Note over DB: Discriminator survives SQLite round-trip

    Note over GW,Model: Next connection / resume
    GW->>DB: get_messages(session_id)
    DB-->>GW: "history tail has finish_reason=interrupt_close"
    GW->>GW: _is_interrupt_close_tail returns True
    GW->>GW: "_build_resume_pending_message returns surface_and_ask=True"
    GW->>GW: "agent._resume_summary_only = True"
    GW->>Model: System note - tell user what completed, ask go-or-redirect

    Model-->>Exec: tool_call emitted ignoring note
    Exec->>Exec: _block_resume_summary_only_tools blocks call
    Note over Exec: RESUME_AUTOCONTINUE_VIOLATION logged
    Exec-->>Model: synthetic error result summarize-only await user go

    Model-->>GW: Text summary asking user

    Note over GW,Model: Next human turn with go message
    GW->>GW: _clear_resume_summary_only_for_human_turn sets False
    GW->>Model: Normal turn tools unblocked

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GW as Gateway
    participant DB as SessionDB
    participant Exec as tool_executor
    participant Model as LLM

    Note over GW,Model: Mid-turn gateway restart
    GW->>DB: "append_message finish_reason=interrupt_close"
    Note over DB: Discriminator survives SQLite round-trip

    Note over GW,Model: Next connection / resume
    GW->>DB: get_messages(session_id)
    DB-->>GW: "history tail has finish_reason=interrupt_close"
    GW->>GW: _is_interrupt_close_tail returns True
    GW->>GW: "_build_resume_pending_message returns surface_and_ask=True"
    GW->>GW: "agent._resume_summary_only = True"
    GW->>Model: System note - tell user what completed, ask go-or-redirect

    Model-->>Exec: tool_call emitted ignoring note
    Exec->>Exec: _block_resume_summary_only_tools blocks call
    Note over Exec: RESUME_AUTOCONTINUE_VIOLATION logged
    Exec-->>Model: synthetic error result summarize-only await user go

    Model-->>GW: Text summary asking user

    Note over GW,Model: Next human turn with go message
    GW->>GW: _clear_resume_summary_only_for_human_turn sets False
    GW->>Model: Normal turn tools unblocked

Loading

_{Reviews (3): Last reviewed commit: "fix(gateway): persist in-place interrupt..." | Re-trigger Greptile}