Add run-journal replay timeline parity checks

[franksong2702]

Thinking Path

• Add WebUI run event journal replay #2283 shipped the first RFC: Make WebUI a thin observability/control client over Hermes Agent runtime #1925 run-journal replay slice: already-emitted SSE events can be replayed after reconnect or a dead WebUI stream.
• Document WebUI run state consistency contract #2363 then documented the run-state consistency contract across transcript, context, streams, journal replay, compression, and the browser timeline.
• Long tool-heavy turns now depend on a structured browser timeline: Thinking, visible progress text, tool cards, compression cards, and terminal states.
• The replay layer should not grow a second renderer or replay those events as a flattened/reordered scene.
• This PR makes that contract explicit and pins it with focused frontend regression coverage.

What Changed

• Added frontend static regression coverage that verifies replay-relevant long-task events enter the same _wireSSE() EventSource handler pipeline as live streaming:
  • reasoning
  • interim_assistant
  • tool
  • tool_complete
  • compressing
  • compressed
  • metering
  • terminal events such as done, apperror, and cancel
• Added coverage that those user-visible long-task events are included in the run-journal cursor tracking loop so replay starts after the latest rendered event instead of duplicating timeline content.
• Clarified the Document WebUI run state consistency contract #2363 run-state consistency RFC: replayed long-task events should use the same browser-facing timeline renderer as live SSE events.
• Added a changelog entry for the contract/test hardening.

Why It Matters

This is the concrete follow-up that connects the shipped run-journal replay slice (#2283) to the run-state consistency contract (#2363) without expanding #2347 or starting the RuntimeAdapter/sidecar work.

The intended architecture is:

live SSE events      ┐
                     ├─ shared EventSource handlers / timeline renderer ─ user-visible timeline
journal replay events┘

Not:

live stream renderer
journal replay renderer

That keeps WebUI thin in execution ownership without making replay a separate presentation/runtime truth.

Verification

• node --check static/messages.js
• git diff --check
• uv run --python 3.12 --with pytest pytest -q tests/test_run_journal_frontend_static.py
  • 5 passed
• uv run --python 3.12 --with pytest pytest -q tests/test_run_journal.py tests/test_run_journal_routes.py tests/test_run_journal_frontend_static.py tests/test_run_journal_streaming_static.py
  • 24 passed

Note: python3 -m pytest ... on this Mac's system Python 3.9 fails while importing current master because the repo uses Python 3.10+ union syntax (str | None). CI targets Python 3.11/3.12/3.13, so verification used Python 3.12 via uv.

Risks / Follow-ups

• This is intentionally a contract/test/doc PR. It does not change runtime behavior.
• It does not make WebUI-owned execution survive a WebUI process restart.
• It does not change Preserve live agent timeline across session switches #2347's live timeline/session-switch behavior.
• If future SSE event names are added, the cursor-tracking event list still needs to be updated. This PR makes that drift visible.
• No before/after screenshots are included because there is no user-visible UI change in this PR.

Refs #2376.
Refs #1925, #2283, #2361, #2363, #2347.

Model Used

OpenAI Codex (GPT-5) assisted with the analysis, implementation, tests, and PR preparation.

[Michaelyklam]

Reviewed this from the #1925 / #2283 replay boundary and it looks like the right narrow contract hardening slice.

What I checked:

• The diff is limited to static regression coverage, the Document WebUI run state consistency contract #2363 consistency RFC note, and the changelog.
• The tests pin the key invariant: replayed long-task events continue through the same _wireSSE() EventSource handler path as live streaming rather than creating a second replay renderer.
• Cursor tracking covers the user-visible long-task events needed to avoid duplicate replay (token, interim_assistant, reasoning, tool/compression events, done, apperror, cancel).
• Scope stays within the accepted production-observation/test-contract lane: no RuntimeAdapter/sidecar, no new runtime ownership claim, and no behavioral rewrite of the live runner.

Local verification from the PR head:

env -u HERMES_CONFIG_PATH -u HERMES_WEBUI_HOST /home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_run_journal_frontend_static.py tests/test_run_journal.py tests/test_run_journal_routes.py tests/test_run_journal_streaming_static.py -q
# 24 passed

git diff --check origin/master...HEAD
# clean

LGTM for the focused #2376 / #1925 follow-up. This should remain a replay parity/regression-contract PR, not the start of slice-2 RuntimeAdapter or runner-survives-restart work.

[nesquena-hermes]

Read the PR worktree against origin/master and the _wireSSE block in static/messages.js. The contract this PR pins down is real and the test shape is the right one.

What the PR actually claims

Two invariants for the replay path:

1. Every long-task SSE event that the live stream renders (reasoning, interim_assistant, tool, tool_complete, compressing, compressed, metering, done, apperror) must flow through the same _wireSSE() handlers when replayed, not a parallel renderer.
2. Every user-visible timeline event must be in the run-journal cursor-tracking loop, otherwise replay re-emits already-rendered events and the timeline duplicates.

Both invariants are property checks on the source — they don't actually run replay, they enforce that the code wiring still goes through the shared handlers. That's the right depth of test for this layer.

Confirming the live handlers exist

static/messages.js:1182 defines _wireSSE(source) and the file has every handler the test enumerates:

1198:    source.addEventListener('token',
1210:    source.addEventListener('interim_assistant',
1236:    source.addEventListener('reasoning',
1254:    source.addEventListener('tool',
1286:    source.addEventListener('tool_complete',
1402:    source.addEventListener('done',
1598:    source.addEventListener('compressing',
1616:    source.addEventListener('compressed',
1643:    source.addEventListener('metering',
1661:    source.addEventListener('apperror',
1759:    source.addEventListener('cancel',

And the cursor loop at static/messages.js:1801 matches the test's expectation:

for(const _runJournalEventName of ['token','interim_assistant','reasoning','tool','tool_complete','approval','clarify','title','title_status','goal','goal_continue','done','stream_end','pending_steer_leftover','compressing','compressed','metering','apperror','warning','error','cancel']){
  source.addEventListener(_runJournalEventName,_rememberRunJournalCursor);
}

The PR's test asserts the subset relevant to long-task timeline parity (drops approval, clarify, title*, goal*, stream_end, warning, error — which are intentional non-timeline events). That's the right scoping: those events have their own re-entry semantics on replay and shouldn't be lumped into a "timeline parity" assertion.

Test shape commentary

wire_pos = MESSAGES_SRC.index("function _wireSSE(source)")
wire_block = MESSAGES_SRC[wire_pos : MESSAGES_SRC.index("async function _restoreSettledSession", wire_pos)]

Slicing on the _restoreSettledSession anchor is brittle in the abstract — if that function ever moves or renames, the test fails loudly. That's fine here because the cursor-tracking loop sits inside _wireSSE and the next function down is in fact _restoreSettledSession. If someone reorganizes the file they'll need to update both the function and this test; that's a feature, not a bug, because it forces explicit acknowledgement of the contract.

Two small refinements that would harden the test further (not blockers for this PR, but worth a follow-up):

1. The _wireSSE block-detection could grep for function _wireSSE then walk balanced braces, rather than relying on the next function name. Pros: survives function reorder. Cons: more complex. I'd leave it as-is given the existing static test culture in this file.
2. assert "updateThinking(" in wire_block and assert "appendLiveToolCard(tc)" in wire_block are good — they pin specific renderer functions, not just the handler registration. That catches the regression where a handler exists but its body got rewired to a separate replay renderer. Useful.

Scope check

The PR diff is exactly the lane the comment from @Michaelyklam describes: regression coverage + RFC clarification + changelog. No _wireSSE body changes, no RuntimeAdapter, no behavioral changes to the live runner. The associated docs/RFC change in docs/rfcs/webui-run-state-consistency-contract.md extends the contract to say "replayed long-task events must use the same browser-facing timeline renderer as live SSE events" — that matches what the tests pin.

Verdict

LGTM. The tests give #2283 (already-merged replay slice) and #2376 (the consistency-tracking issue) a real ratchet: if a future slice-2 RuntimeAdapter PR tries to introduce a parallel replay renderer, these tests fail. That's the right shape for a "before we land more replay infra, lock the contract" PR.

One nit: in test_run_journal_cursor_tracks_every_long_task_timeline_event, the 700-byte window after the cursor_loop_pos index is a magic number. If the cursor-loop list ever grows past that, the assertion silently misses entries past byte 700. A safer bound would be cursor_loop = MESSAGES_SRC[cursor_loop_pos : MESSAGES_SRC.index(']', cursor_loop_pos)]. Trivially fixable in a follow-up.

[franksong2702]

Follow-up PR #2390 is now open as the live-behavior companion to this replay parity contract.

#2377 is still the right narrow contract/test PR: replayed long-task events should go through the same _wireSSE() timeline path and cursor tracking as live streaming.

The new finding from real 8787 validation was one layer below that contract: live streaming itself still had a boundary bug where a tool start could close the current Activity group. That meant a valid backend sequence like:

interim_assistant: visible progress note
tool: terminal
tool_complete: terminal
tool: terminal
tool_complete: terminal

could render as repeated one-tool Activity rows instead of one grouped Activity row.

#2390 fixes that live grouping invariant:

• visible interim_assistant text closes the previous Activity burst;
• hidden reasoning updates do not split tool bursts;
• tool starts reset the next assistant text segment but do not close the current Activity group.

So the relationship is:

• Add run-journal replay timeline parity checks #2377: replay must preserve the live long-task timeline contract.
• Fix live progress Activity grouping #2390: live long-task timeline grouping must itself use the right progress/tool boundaries.

[franksong2702]

Follow-up issue #2423 is open for the next replay/restart gap found in real validation.

This PR correctly pinned the replay contract: journaled long-task events should go through the same _wireSSE() timeline handlers and cursor tracking as live events.

The new issue is not that the replay endpoint failed. It worked: the stale stream reported replay_available=true and /api/chat/stream?...&replay=1 returned journaled events.

The gap is higher in the reload/repair path: after WebUI process restart, the stale stream is cleared and an interrupted marker is written to the session, but the already-journaled partial timeline is not surfaced to the user.

That makes #2423 a natural follow-up to this contract PR: replay exists, but restart recovery needs to use it before telling the user "no agent output was recovered."