Fix #5532: /api/session/clear sets truncation watermark (P0 data loss)

[nesquena-hermes]

Summary

Closes #5532 (P0 data loss).

POST /api/session/clear cleared the sidecar transcript but never recorded a
truncation watermark, so state.db messages resurrected on the next
/api/session read: history reappeared after clear + refresh, and a continued
conversation still carried the full pre-clear context into the model.

Root cause

The /api/session/clear handler (api/routes.py) did:

s.messages = []
s.tool_calls = []
apply_session_title_rename(s, "Untitled")
s.save()

It never set s.truncation_watermark. The append-only state.db merge
(merge_session_messages_append_only, reached via
reconciled_state_db_messages_for_session) treats an unset / None watermark
as "keep everything and just dedup". Because state.db is append-only, the
cleared transcript was still there, so the merge re-added all of it on the next
read — the exact data-loss symptom in the issue.

The sibling destructive op /api/session/truncate does not have this bug:
it calls truncate_session_at_keep(session, keep) in api/session_ops.py,
which sets truncation_watermark = truncation_boundary = _truncation_watermark_for(kept_messages).

Fix

A full clear is just a truncate that keeps zero messages. Route /clear
through the same helper the truncate handler uses:

truncate_session_at_keep(s, 0)   # empties messages + context_messages,
                                 # sets watermark = boundary = 0.0
s.tool_calls = []
apply_session_title_rename(s, "Untitled")
s.save()

_truncation_watermark_for([]) == 0.0, which is the #2914 "truncate-to-empty"
sentinel that blocks all state.db replay. The merge/read path is
untouched — /clear and /truncate now set the marker identically, so the
merge contract is not forked.

Second half of the issue (continued-context) — verified

The issue notes that a continued conversation retained pre-clear context. That
is fixed by the same change: truncate_session_at_keep(s, 0) empties
context_messages in lockstep with messages (via
truncate_context_for_display_keep(..., keep=0) -> []), so the model-facing
context no longer carries the cleared turns. A regression test asserts
context_messages == [] after clear.

Tests

tests/test_issue5532_clear_truncation_watermark.py (drives the real
POST /api/session/clear route via handle_post, mirroring the #2914
integration harness):

• test_clear_endpoint_sets_truncation_watermark — watermark & boundary set to 0.0
• test_clear_empties_context_messages — second-half fix: context_messages == []
• test_clear_then_read_does_not_resurrect_state_db_messages — subsequent
  state.db merge returns EMPTY (no resurrection)
• test_clear_matches_truncate_to_empty_marker — /clear and
  /truncate(keep=0) produce an identical marker (no forked merge logic)

Fail-without-fix confirmed: all four fail on origin/master
(truncation_watermark stays None, resurrection occurs) and pass with this
change.

$ .venv/bin/python3 -m pytest tests/test_issue5532_clear_truncation_watermark.py -q
....                                                                     [100%]
4 passed in 3.80s

Related watermark suite stays green:

$ .venv/bin/python3 -m pytest tests/test_issue5532_clear_truncation_watermark.py \
    tests/test_issue2914_truncation_watermark.py tests/test_issue3831_watermark_clear.py -q
30 passed

[greptile-apps]

Greptile Summary

Routes POST /api/session/clear through the existing truncate_session_at_keep(s, 0) helper so it sets truncation_watermark = truncation_boundary = 0.0, closing the gap that caused cleared messages to resurrect from the append-only state.db on the next read.

• Core fix (api/routes.py): replace the manual s.messages = [] assignment with truncate_session_at_keep(s, 0), which also empties context_messages and stamps the truncation watermark that the merge path requires to honor the clear.
• Codex gates (api/routes.py): two follow-on hazards are closed — compression-lineage detachment (nulls parent_session_id + anchor fields so a compressed-continuation parent snapshot can't resurface the pre-clear history on display), and stale-backup removal (s.path.with_suffix(".json.bak").unlink(missing_ok=True) so startup recovery can't undo the clear across a restart).
• Tests (tests/test_issue5532_clear_truncation_watermark.py): six regression tests drive the real handle_post route end-to-end and cover watermark presence, context_messages erasure, no state.db resurrection, marker parity with /truncate, compression-lineage detachment, and backup-survival protection.

Confidence Score: 5/5

Safe to merge — the change is a targeted rerouting of /clear through an already-proven helper, with no modifications to the merge/read path itself.

The fix is minimal and mechanically sound: truncate_session_at_keep(s, 0) is the same code path already exercised by /truncate, so no new logic is introduced for the critical merge contract. The three follow-on hazards (watermark, compression lineage, stale backup) are each independently tested end-to-end against the real route handler. RFC line-number drift is the only other change and is a documentation-only update with consistent +46 offsets throughout.

No files require special attention.

Important Files Changed

Filename	Overview
api/routes.py	Replaces the bare s.messages = [] assignment in /api/session/clear with truncate_session_at_keep(s, 0) and adds compression-lineage nulling and stale-backup removal; logic is correct and mirrors the sibling /truncate handler.
tests/test_issue5532_clear_truncation_watermark.py	Six end-to-end regression tests exercising the real handle_post route; harness pattern mirrors existing #2914 suite, stubs are minimal and correct (CSRF, eviction), assertions cover all failure modes described in the issue.
docs/rfcs/session-sse-contract-v1.md	Mechanically updates route.py line-number references (+46 across all anchors, consistent with the 46-line expansion of the clear handler); no semantic content changed.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant C as Client
    participant R as routes.py /api/session/clear
    participant SO as session_ops.truncate_session_at_keep
    participant S as Session object
    participant DB as state.db (append-only)
    participant BAK as .json.bak
    participant REC as session_recovery

    C->>R: "POST /api/session/clear {session_id}"
    R->>SO: "truncate_session_at_keep(s, keep=0)"
    SO->>S: "s.messages = []"
    SO->>S: "s.context_messages = []"
    SO->>S: "s.truncation_watermark = 0.0 (sentinel)"
    SO->>S: "s.truncation_boundary = 0.0"
    R->>S: "s.tool_calls = []"
    R->>S: "s.parent_session_id = None (detach compression lineage)"
    R->>S: "s.compression_anchor_* = None"
    R->>S: apply_session_title_rename(s, Untitled)
    R->>S: s.save()
    R->>BAK: "unlink(.json.bak, missing_ok=True)"
    Note over BAK: Prevents startup recovery from resurrecting cleared transcript

    C->>R: GET /api/session (next read)
    R->>DB: reconciled_state_db_messages_for_session(s)
    Note over DB: watermark=0.0 blocks ALL replay
    DB-->>R: [] (empty — no resurrection)
    R-->>C: "{messages: []}"

    REC->>BAK: check .json.bak on startup
    Note over REC: .bak is gone → no restore attempt

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant C as Client
    participant R as routes.py /api/session/clear
    participant SO as session_ops.truncate_session_at_keep
    participant S as Session object
    participant DB as state.db (append-only)
    participant BAK as .json.bak
    participant REC as session_recovery

    C->>R: "POST /api/session/clear {session_id}"
    R->>SO: "truncate_session_at_keep(s, keep=0)"
    SO->>S: "s.messages = []"
    SO->>S: "s.context_messages = []"
    SO->>S: "s.truncation_watermark = 0.0 (sentinel)"
    SO->>S: "s.truncation_boundary = 0.0"
    R->>S: "s.tool_calls = []"
    R->>S: "s.parent_session_id = None (detach compression lineage)"
    R->>S: "s.compression_anchor_* = None"
    R->>S: apply_session_title_rename(s, Untitled)
    R->>S: s.save()
    R->>BAK: "unlink(.json.bak, missing_ok=True)"
    Note over BAK: Prevents startup recovery from resurrecting cleared transcript

    C->>R: GET /api/session (next read)
    R->>DB: reconciled_state_db_messages_for_session(s)
    Note over DB: watermark=0.0 blocks ALL replay
    DB-->>R: [] (empty — no resurrection)
    R-->>C: "{messages: []}"

    REC->>BAK: check .json.bak on startup
    Note over REC: .bak is gone → no restore attempt

Loading

_{Reviews (4): Last reviewed commit: "fix(#5532): resolve CI — ruff F841 + rea..." | Re-trigger Greptile}

[nesquena-hermes]

Cross-ref for merge ordering: #5569 (de-brittle #5542) converts the SSE-contract RFC to cite api/routes.py symbols by name instead of :NNNN line numbers, and removes the line anchors entirely.

This PR (#5553) currently includes a small RFC hunk that realigns those line numbers (+46, to survive this PR's routes.py insertion) so the pre-#5569 line-anchor test stays green. That realignment becomes obsolete once #5569 lands.

• If fix(#5542): de-brittle SSE-contract RFC anchors — cite by symbol, not line number #5569 merges first: this PR's RFC hunk will conflict trivially — resolve by dropping the line-number realignment (the anchors no longer exist; the symbol-based test doesn't care about this PR's line shift).
• If Fix #5532: /api/session/clear sets truncation watermark (P0 data loss) #5553 merges first: fix(#5542): de-brittle SSE-contract RFC anchors — cite by symbol, not line number #5569 cleanly deletes the (realigned) line numbers.

No code conflict either way — both only touch the RFC's citation style. The P0 code fix in this PR is independent of #5569.

[nesquena-hermes]

Shipped in v0.51.860 🎉 as part of the combined #5532 fix (primary PR #5556 by @rodboev).

Your improvements were folded in and credited (Co-authored-by): the shared truncate_session_at_keep(s,0) helper-reuse and — the key Codex-caught deeper path — detaching the pre_compression_snapshot parent lineage so a compressed-continuation child can't resurrect its pre-clear transcript via the parent display-stitch (which merges with watermark=None, bypassing the child's 0.0 sentinel). Guarded to detach only compression-snapshot parents so ordinary fork links survive. Both your test file and #5556's ship. Closing as combined-and-shipped — thank you.