fix(#5532): prevent clear from replaying state.db history

[rodboev]

Thinking Path

• /api/session/clear should be a destructive reset, but it currently clears only the WebUI sidecar arrays and leaves the persistent merge unmarked.
• The display merge already has the right mechanism: the truncate-to-empty sentinel blocks replay from state.db when a sidecar was intentionally emptied.
• The same route also needs to clear context_messages, because provider-facing history can otherwise keep pre-clear turns even when the visible transcript is empty.
• A clear that shrinks a non-empty sidecar can create a recovery backup, so the route removes only that clear-created backup while preserving unrelated recovery backups.

What Changed

• api/routes.py: clear context_messages, stale active_stream_id, and pending user-message fields alongside messages and tool_calls, persist the existing truncate-to-empty sentinel, and remove the route-created pre-clear .json.bak only after the fully cleared state is verified on disk.
• tests/test_issue5532_session_clear_state_db_replay.py: cover clear plus state.db replay suppression, context_messages reset, stale active/pending state reset, route-created backup removal after verified persistence, existing backup preservation for already-empty sidecars, later backup recovery, and normal un-cleared state.db recovery.
• docs/rfcs/session-sse-contract-v1.md: refresh route source anchors shifted by the api/routes.py insertion.

Why It Matters

Clear Conversation now stays cleared after refresh, restart recovery, and the next prompt. Users no longer get old transcript rows or hidden model context after asking WebUI to reset a session.

Verification

• python -m pytest tests/test_issue5532_session_clear_state_db_replay.py tests/test_3230_preserve_manual_session_title.py::test_clear_route_uses_rename_helper_not_bare_title_assignment tests/test_memory_session_lifecycle_generation.py::test_clear_session_evicts_outside_session_lock tests/test_session_truncate_keep_count_validation.py::test_truncate_zero_keep_count_clears_transcript tests/test_issue4836_manual_compression_recovery.py tests/test_metadata_save_wipe_1558.py -v --timeout=60
• python -m pytest tests/test_issue4812_session_sse_contract_rfc.py::TestEndpointDistinction::test_rfc_cites_current_global_endpoint_source tests/test_issue4812_session_sse_contract_rfc.py::TestEndpointDistinction::test_rfc_run_journal_anchors_land_on_real_source -v --timeout=60

Full-suite CI context, not a required local check: python -m pytest tests/ -v --timeout=60.

Upstream

Closes #5532.

The maintainer confirmation at #5532 (comment) shaped the sentinel approach and the context_messages reset.

Related: #5553 covers the route sentinel and context reset path. This branch also guards the clear-created .json.bak recovery edge so restart recovery cannot undo the clear, while preserving unrelated backup recovery.

Model Used

GPT 5.5 via Codex CLI

[greptile-apps]

Greptile Summary

This PR fixes /api/session/clear so that clearing a session persists durably and cannot be undone by restart recovery or state.db replay. The core change zeros out context_messages, all pending fields, and the truncation watermarks before calling save() — which also has the side effect of satisfying the #1558 guard.

• api/routes.py: extends the clear route to zero context_messages, pending fields, and watermarks, then verifies the persisted state after save() and conditionally removes any backup file the save created.
• tests/test_issue5532_session_clear_state_db_replay.py: new regression suite covering replay suppression, context_messages reset, and backup lifecycle.
• docs/rfcs/session-sse-contract-v1.md: updates line-number anchors shifted by the routes.py insertion.

Confidence Score: 5/5

Safe to merge — the clear route now correctly zeroes every field that could survive a restart, the #1558 silent-refusal guard is neutralised by nulling active_stream_id/pending_user_message before save(), and the backup removal is gated on hard verification of the persisted state.

All three layers of the fix (context_messages reset, guard bypass, backup lifecycle) are logically consistent with each other and with the existing save() contract. The test suite exercises the primary happy path, the already-empty edge case, the negative-space case, and later post-clear backup recovery.

No files require special attention. The routes.py change is the only functional change and its logic has been verified against the save() guard in models.py.

Important Files Changed

Filename	Overview
api/routes.py	Clear route extended to zero context_messages and all pending fields, bypassing the #1558 guard; backup removal logic is correctly gated on had_sidecar_messages AND verified persisted state.
tests/test_issue5532_session_clear_state_db_replay.py	New regression suite covering the happy path, negative-space (watermark=None allows replay), already-empty-sidecar preservation, and post-clear backup recovery.
docs/rfcs/session-sse-contract-v1.md	Line-number anchors updated to reflect the 30-line insertion in routes.py; no functional changes.

_{Reviews (2): Last reviewed commit: "fix(#5532): prevent clear from replaying..." | Re-trigger Greptile}

[nesquena-hermes]

🔬 Gate certification — RED ⛔ (direct-history clear is thorough, but a compressed continuation still replays PARENT history via lineage — 1 SILENT)

Certified head: sha:ea2dc31c (rebased onto current master, git apply clean) · PR: #5556 · rodboev, fix(#5532): prevent clear from replaying state.db history
Verdict: The direct-history clear is well-built (verifies the sidecar is fully empty before dropping the backup, fails closed) and the suite is fully green — but Codex found (and empirically verified) that clearing a compressed continuation still replays the PARENT session's history on reload, because the clear doesn't clear/verify parent_session_id and GET /api/session rebuilds display from the parent lineage. Incomplete fix for the same #5532 bug class.

What I ran (rebased worktree /tmp/wt-rebase-5556)

Gate	Result
Rebase onto current master	✅ git apply clean
Codex (reproduce)	SHIP-WITH-FIXES — 1 SILENT (parent-lineage replay); Codex verified with an isolated temp-session runtime check; I confirmed the code path
Full pytest suite	✅ 11989 passed, 0 failed (the RFC's line-pin re-bump kept test_issue4812 green — confirming those pins are the #5513 collateral, root-cause in #5542)
PR's own test	✅ 4/4 (test_issue5532_session_clear_state_db_replay.py)

Findings

⛔ SILENT (I + Codex CONFIRMED) — clearing a compressed continuation replays the PARENT's history (api/routes.py /api/session/clear ~13718): the new persisted_clear verifies the sidecar's own fields are empty (messages, context_messages, truncation watermarks, active_stream_id, pending_*) — thorough — BUT it does NOT clear or verify parent_session_id. GET /api/session builds display via _webui_sidecar_lineage_messages_for_display() / _merged_webui_lineage_messages_for_display() (routes.py:8021/8106/11693), which reintroduce messages from parent_session_id. Codex verified empirically: an empty child with truncation_watermark=0.0 and a pre-compression parent still returned the parent prompt on reload. So a user who clears a compressed continuation still sees the old history come back — the exact #5532 symptom, one path over. Fix (Codex-exact): in /api/session/clear set s.parent_session_id = None before s.save(), add persisted.get("parent_session_id") is None to the persisted_clear predicate, and add a regression test for clearing a child whose parent sidecar is pre_compression_snapshot=True.

✅ Direct-history clear is sound (keep): the fully-empty persisted_clear verification + fails-closed backup removal (had_sidecar_messages and persisted_clear) is the right shape; 4/4 tests. ℹ️ RFC-doc change: just re-bumps the brittle api/routes.py:NNNN line-pins (16177→16207, etc.) to match this PR's routes.py shifts — a stopgap that kept test_issue4812 green (root-cause de-brittle tracked in #5542); no contract-semantics change.

Recommendation to the next agent / author

RED — gate-fail/changes-requested (1 SILENT): extend the clear to the lineage path — s.parent_session_id = None before save + add persisted.get("parent_session_id") is None to persisted_clear + a compressed-continuation regression test. The direct-history fix is thorough and the suite is green; this one gap (parent-lineage replay) is the same #5532 bug via a different path — a "clear" must sever the parent lineage too, or a compressed continuation replays. concept 4/5 (real data-integrity fix; one lineage path missed). Author @rodboev (T1). crit=3, data-integrity. (Gate value: same class as #5504 — a data-clear/tombstone must cover EVERY path that can reintroduce the data; here the sidecar-own fields were verified but the parent-lineage display path wasn't. When clearing/hiding history, enumerate all readers that reconstruct it — lineage/parent, backup, state.db, cache.)

---

_Gate-certifier layer (warm-up → gate → release). I do not merge/tag/deploy. Rebased onto current master; direct-history clear verified thorough (fully-empty persisted_clear, fails closed) BUT clear route touches ZERO parent_session_id (grep empty) → compressed-continuation replays parent history via _*lineage_messages_for_display (Codex empirical temp-session check); suite fully green (RFC line-bump kept #4812 green, #5513 collateral per #5542). Cert valid for sha:ea2dc31c.

[nesquena-hermes]

Shipped in v0.51.860 🎉 — merged via release stage-5532b.

Thanks @rodboev — this closes the #5532 P0 (clearing a conversation didn't stick: history resurrected from state.db after a refresh, and continued chats carried the full pre-clear context to the model). Your fix (persisted-clear verification + stale-backup removal + the truncate-to-empty watermark) shipped as the primary, combined with the sibling maintainer PR #5553's improvements per the reliability review:

• routes clear through the shared truncate_session_at_keep(s,0) helper (single source of truth with /api/session/truncate)
• detaches the compression-snapshot parent link so a compressed-continuation child can't resurrect its transcript via the parent display-stitch — guarded so genuine fork parent links (nesting + "Forked from") are preserved

Gate: Codex SAFE + Opus SHIP + full suite 11997 passed. Non-blocking follow-ups filed: #5570, #5571, #5572.