fix(todos): stream live state updates with inflight recovery

[v2psv]

Bug Description

Follow-up to the cold-load todo state work from #3373.

The Todos panel can now hydrate correctly after a page refresh, but it still does not consistently track the current todo tool state while an agent run is active.

Before this PR:

• todo tool calls updated the agent-side TodoStore, but the side panel usually stayed stale until the run settled.
• A browser reload / SSE reattach during an active stream could temporarily lose or roll back the visible todo snapshot.
• Cold-load session state and persisted INFLIGHT state did not share one explicit reconciliation path, so stale local recovery data could win over the settled session snapshot in some edge cases.

Root Cause

The panel historically derived todo state by reverse-scanning settled tool messages in S.messages.

That works after a completed turn, but not during a live run:

• live tool completions are tracked in stream / INFLIGHT structures before settled tool messages appear in S.messages;
• tool_complete.preview is truncated and not a durable structured state contract;
• cold-load session hydration and INFLIGHT recovery had no single source of truth or timestamp reconciliation rule.

Fix

This PR adds an explicit todo-state contract across server, SSE, and frontend state:

• Emits a dedicated todo_state SSE event when the todo tool completes.
• Sends full, redacted, idempotent todo snapshots instead of relying on truncated tool_complete.preview.
• Reuses the same api.todo_state parser / normalizer for live emission and cold-load state.
• Updates the frontend to treat S.todos + S.todoStateMeta as the Todos panel source of truth.
• Persists live todo snapshots into INFLIGHT state so reload / reattach can restore the panel before the next event arrives.
• Reconciles cold-load vs INFLIGHT snapshots by timestamp, including the coldTs === 0 edge case for compressed sessions.
• Clears stale session.todo_state sidecars when a fresh messages=1 response does not include one.
• Keeps the legacy reverse-scan fallback for old servers / upgrade windows.
• Adds a CHANGELOG entry for the user-visible behavior fix.

How to Verify

Manual verification path:

1. Start an agent run that calls the todo tool multiple times.
2. Keep the Todos panel open while the stream is still active.
3. Confirm the panel updates after each todo snapshot instead of waiting for the final assistant response.
4. Reload / reattach during the active stream.
5. Confirm the panel restores the current todo snapshot from INFLIGHT and does not flicker back to an older list.
6. Open the same session cold after the run settles.
7. Confirm the panel hydrates from the server-provided session.todo_state.

Test Plan

Ran locally after rebasing onto origin/master:

/root/hermes-webui/.venv/bin/python -m pytest \
  tests/test_todo_state.py \
  tests/test_todo_state_emission.py \
  tests/test_streaming_todo_state_static.py \
  tests/test_session_todo_state_route.py \
  tests/test_todo_live_frontend_static.py \
  tests/test_todo_panel_cold_load_static.py \
  tests/test_security_redaction.py \
  -q

Result:

83 passed, 1 skipped

Also ran:

/root/hermes-webui/.venv/bin/python scripts/ruff_lint.py --diff origin/master

Result:

ruff_lint: no new violations on added/modified lines. OK.

Additional local hygiene checks:

git diff --check origin/master..HEAD

No whitespace errors or conflict markers found.

Risk Assessment

Low / Medium.

The change is scoped to Todos panel state propagation and recovery. The new SSE event is additive, and the frontend keeps the legacy message-scan fallback for compatibility with older servers.

Risk is mainly in state reconciliation:

out-of-order SSE replay is guarded by timestamp checks;
cross-session events are filtered by session_id;
malformed or non-todo payloads are ignored;
live SSE snapshots are redacted before emission;
cold-load snapshots continue to pass through session response redaction.
The PR intentionally keeps the detector symmetric with the existing agent TodoStore hydration logic so the WebUI panel does not disagree with the agent-side state.

[nesquena-hermes]

Reading the full diff at 888db2b5 — api/todo_state.py, api/streaming.py, and the four frontend files — this is a well-structured change. The single-source-of-truth model (S.todos + S.todoStateMeta sentinel in static/ui.js, fed by the todo_state SSE listener / INFLIGHT restore / cold-load), the latest-by-position vs. recency reconciliation in _hydrateTodosFromSession, and the fail-closed redaction on the live path all line up with the agent's _hydrate_todo_store contract. I verified that against the agent side: run_agent.py:2670-2699 walks history in reverse, breaks on the first role='tool' message with a todos list, and crucially does if last_todo_response: — so an empty [] write leaves the store empty. Your _normalize_snapshot docstring and the _legacyTodosFromMessages change (dropping the d.todos.length guard) keep the panel symmetric with that. Good.

One thing I'd flag before merge — a hot-path perf regression in the redaction helper.

Code reference

api/todo_state.py:213-236 (PR HEAD):

def _redact_snapshot(snapshot: dict) -> dict:
    from typing import cast
    from api.helpers import _redact_value
    return cast(dict, _redact_value(snapshot))

_redact_value is called without the _enabled kwarg, so it defaults to None. Tracing into api/helpers.py:354-366 → _redact_text at helpers.py:334-351:

def _redact_text(text, *, _enabled=None):
    if _enabled is None:
        from api.config import load_settings
        _enabled = bool(load_settings().get("api_redact_enabled", True))

So every string in the snapshot triggers its own load_settings(). And load_settings() (api/config.py:5039) is not memoized — each call does SETTINGS_FILE.read_text() plus a get_config() resolve.

Why it matters here

emit_todo_state fires on the live streaming path, in both tool-callback shapes (api/streaming.py:4777 and :4874). A todo snapshot has ~3 strings per item (id, content, status), so a 20-item list = ~60 load_settings() disk reads per todo write. A planning-heavy run with 10-30 todo calls multiplies that into hundreds-to-thousands of settings.json reads across one turn — on the SSE emit path, which is exactly where the existing "Opus pre-release perf fix" added the _enabled threading to redact_session_data (helpers.py:369-391) to avoid.

Recommendation

Thread the setting once, mirroring redact_session_data:

def _redact_snapshot(snapshot: dict) -> dict:
    from typing import cast
    from api.helpers import _redact_value
    from api.config import load_settings
    _enabled = bool(load_settings().get("api_redact_enabled", True))
    return cast(dict, _redact_value(snapshot, _enabled=_enabled))

That's one read per emission instead of one per string, and it keeps the fail-closed behavior intact (a raising load_settings() still propagates out of emit_todo_state's try and drops the event).

Test note

tests/test_todo_state_emission.py::test_emit_todo_state_redacts_before_sse monkeypatches helpers._redact_value directly, so it won't catch the _enabled threading either way — the fix above keeps that test green. If you want a guard against the regression returning, an ast-based assertion (like tests/test_streaming_todo_state_static.py already does for the emit calls) that _redact_snapshot passes _enabled= would lock it in.

Everything else looks solid — CI is green across the 3.11/3.12/3.13 matrix and the cross-session session_id/activeSid double-filter on the SSE listener matches the other live listeners.

[nesquena-hermes]

Re-read api/todo_state.py:213-240 and tests/test_todo_state_emission.py at the new HEAD — the perf regression I flagged is resolved correctly.

_redact_snapshot now reads the setting once and threads it through the recursive helper:

from api.config import load_settings
from api.helpers import _redact_value
_enabled = bool(load_settings().get("api_redact_enabled", True))
return cast(dict, _redact_value(snapshot, _enabled=_enabled))

That's one settings.json read per SSE emission instead of one per string, and the fail-closed behavior is preserved — a raising load_settings() still propagates out of emit_todo_state's try and drops the event.

The new test_emit_todo_state_threads_redaction_enabled_flag_once locks it in well: it asserts settings_calls == ["load_settings"] (exactly one read for a multi-string snapshot, which would have been 3+ on the old per-string path) and redaction_enabled_args == [False] (the flag is actually threaded, not re-derived). The existing test_emit_todo_state_redacts_before_sse was also updated to the *, _enabled=None signature so it stays green.

No further concerns from me on the redaction path. LGTM.

[greptile-apps]

Greptile Summary

This PR adds a dedicated todo_state SSE event so the Todos panel updates in real-time during active agent runs, and introduces a unified reconciliation path (_hydrateTodosFromSession) that picks the fresher of the cold-load server snapshot or the persisted INFLIGHT snapshot on reload/reattach.

• Backend (api/todo_state.py, api/streaming.py): emit_todo_state() fires a redacted, full-snapshot todo_state SSE event at each todo tool completion; two call sites cover the legacy-preview and modern-result callback shapes.
• Frontend (static/ui.js, static/messages.js, static/sessions.js, static/panels.js): S.todos / S.todoStateMeta become the single source of truth for the panel; _hydrateTodosFromSession reconciles cold-load vs. INFLIGHT by timestamp (with a streamActive tie-break for timestamp-less snapshots); the legacy message-scan fallback is retained for older servers.
• api/models.py: Two ValueError guards are added in all_sessions() to detect index/file inconsistencies, though one guard can fire in legitimate post-cleanup states.

Confidence Score: 3/5

The todo state propagation path is well-designed with idempotent snapshots, timestamp guards, and redaction, but two call sites that update S.todos do not schedule a panel render, and a new ValueError in all_sessions() can surface as a 500 on the sessions list in post-cleanup edge cases.

The core SSE emission and reconciliation logic is solid and the test coverage is unusually thorough for frontend state work. newSession() calls _hydrateTodosFromSession but not scheduleTodosRefresh(), so the Todos panel visibly lags when creating a new session with the panel open. The new ValueError guards in all_sessions() are asymmetric — _session_dir_has_persisted_session_files() checks for any json file while the live-filter uses persisted_ids, so a cleanup-lag state can cause the sessions list endpoint to throw rather than return a partial or empty result.

api/models.py (ValueError guard), static/sessions.js (newSession hydration path), static/messages.js (three stream-refresh handlers missing scheduleTodosRefresh)

Important Files Changed

Filename	Overview
api/models.py	Adds two defensive ValueError guards in all_sessions() after the session live-filter; the post-filter guard can fire on a legitimate cleanup-lag state and break the sessions list endpoint.
api/streaming.py	Wires emit_todo_state() into both tool-callback shapes (legacy preview fallback and modern result path); error-swallow policy keeps tool delivery unaffected.
api/todo_state.py	Major expansion: adds emit_todo_state, _redact_snapshot, and moves derive_todo_state; logic is solid but _redact_snapshot reads load_settings() on every SSE emission.
static/messages.js	Adds todo_state SSE listener with cross-session and timestamp guards; correctly persists to INFLIGHT. Three session-refresh handlers call _hydrateTodosFromSession but omit scheduleTodosRefresh(), leaving the panel potentially stale after compression events.
static/panels.js	Refactors loadTodos() to read from S.todos/S.todoStateMeta; adds hash-based short-circuit and RAF coalescing; legacy fallback preserved for old servers.
static/sessions.js	Plumbs _hydrateTodosFromSession into loadSession/newSession/delete paths and restores todos from INFLIGHT; newSession omits scheduleTodosRefresh(), leaving the Todos panel stale after session creation if the panel is open.
static/ui.js	Adds todos/todoStateMeta to global S state, implements _hydrateTodosFromSession with cold-vs-INFLIGHT timestamp reconciliation, and adds scheduleTodosRefresh RAF coalescing; logic is well-thought-out.
tests/test_todo_state_emission.py	New tests cover emit_todo_state happy path, invalid/non-todo payloads, redaction integration, and settings-call count.
tests/test_streaming_todo_state_static.py	Static AST-level tests verify that streaming.py imports emit_todo_state, calls it in both callback shapes, and prefers full result over preview.
tests/test_todo_live_frontend_static.py	Thorough static and node-executed tests for the frontend SSE listener: validates session filtering, timestamp ordering, INFLIGHT mirroring, and malformed-payload handling.
tests/test_todo_panel_cold_load_static.py	Updates existing cold-load static tests to match the new _hydrateTodosFromSession/scheduleTodosRefresh API and S.todoStateMeta-based render path.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant streaming as streaming.py
    participant todo_state as todo_state.py
    participant SSE
    participant messages as messages.js
    participant ui as ui.js
    participant panels as panels.js

    Agent->>streaming: todo tool completes
    streaming->>todo_state: emit_todo_state(put, name, result, session_id)
    todo_state->>todo_state: parse_todo_tool_result()
    todo_state->>todo_state: _redact_snapshot()
    todo_state->>SSE: put todo_state event

    SSE->>messages: addEventListener todo_state fires
    messages->>messages: filter by session_id and ts guard
    messages->>ui: S.todos and S.todoStateMeta updated
    messages->>messages: INFLIGHT persisted
    messages->>ui: scheduleTodosRefresh()
    ui->>panels: loadTodos() via RAF

    Note over messages,ui: On session load or reload
    messages->>ui: _hydrateTodosFromSession(session)
    ui->>ui: reconcile cold-load vs INFLIGHT by timestamp
    ui->>ui: _resetTodosRenderCache()

Loading

Comments Outside Diff (2)

1. static/sessions.js, line 650-652 (link)

   Todos panel not refreshed after newSession() hydration

   _hydrateTodosFromSession(S.session) correctly clears S.todos and S.todoStateMeta for a fresh session, but scheduleTodosRefresh() is never called afterwards. renderMessages() on line 669 re-renders the message thread but does not invoke loadTodos(). If the Todos panel is open while switching to a new session, it will continue to show the previous session's todo list until the user manually triggers a refresh. Compare with _ensureMessagesLoaded(), which explicitly follows _hydrateTodosFromSession with scheduleTodosRefresh() — the same pattern is needed here.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

2. api/todo_state.py, line 362-368 (link)

   load_settings() called per SSE emission in _redact_snapshot()

   _redact_snapshot() re-reads settings via load_settings() on every call — once per todo_state SSE event. For an agent invoking todo frequently, this reads and parses the settings file repeatedly in the hot streaming path. Consider caching the _enabled flag at the call site in emit_todo_state and passing it as an argument, so settings reads are bounded per-turn rather than per-tool-call.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "fix(todos): stream live state updates wi..." | Re-trigger Greptile}

[greptile-apps]

ValueError can fire in post-cleanup races, breaking the sessions API

_session_dir_has_persisted_session_files() scans SESSION_DIR for any non-underscore-prefixed .json file, while the filter retains only index entries whose session_id appears in in_memory_ids or persisted_ids. These two predicates are not symmetric: a file for session C can make the helper return True while the index contains only stale entries for sessions A and B (just purged from both stores). After the filter removes A and B, the second guard fires and raises ValueError("session index has no live rows while session files exist") — even though the state is entirely consistent (A/B are legitimately gone; C is simply not yet indexed). This ValueError would surface as a 500 on the sessions list endpoint rather than returning an empty list or triggering the index-rebuild fallback below this block.

[greptile-apps]

_hydrateTodosFromSession calls in stream session-refresh paths lack scheduleTodosRefresh()

The three stream settle / session-refresh handlers added in messages.js (around lines 2136, 2566, and 2664) all call _hydrateTodosFromSession(S.session) but none follow with scheduleTodosRefresh(). If the reconciled snapshot differs from what the panel last rendered (e.g., after a context-compression refresh), the Todos panel stays stale for the remainder of the turn if no further todo calls happen. _ensureMessagesLoaded() sets the expected pattern — all _hydrateTodosFromSession call sites outside the SSE todo_state listener should schedule a refresh.

[nesquena-hermes]

Shipped in v0.51.261 (Release IC) — thank you @v2psv! 🙏 Live Todos panel via the new todo_state SSE contract (full redacted snapshots, INFLIGHT persistence for reload/reattach, timestamp reconciliation, legacy reverse-scan fallback retained). Full suite 7692/0, Codex SAFE (verified SSE handler composition, INFLIGHT cleanup, reconciliation can't let a stale snapshot win, redaction, and no CLI-classification interaction). Closing as merged-via-release-stage.