fix(#6611): make regeneration start atomic

[rodboev]

Thinking Path

• Regeneration must replace the existing logical user turn while preserving the session's canonical transcript and stream ownership.
• A rejected regeneration must not restore state captured before the session lock, because a concurrent send may have become the current owner.
• Imported sessions retain their original CLI, TUI, or Desktop provenance, so the final selected row needs its own durable WebUI ownership marker.
• The server now keeps rollback inside the locked transaction and accepts imported regeneration only for a selected row carrying the private WebUI turn token.
• Public projections replace that private token with _active_turn_user: true, and browser reconciliation now consumes that marker before timestamp fallback.

What Changed

• api/routes.py removes request-time regeneration rollback and prevents imported-message refresh from replacing the sidecar transcript during regeneration preview. GET revision issuance uses the shared authority after a cheap imported-marker check.
• api/session_ops.py centralizes selected-turn ownership. Writable imported sessions require a valid _active_turn_token; read-only sessions, unmarked foreign turns, malformed markers, and invalid fork lineage remain rejected.
• static/ui.js recognizes the projected _active_turn_user marker when restoring a pending active prompt, while retaining private-token and precision timestamp matching for other transcript shapes.
• Focused tests cover concurrent lock winners in both orderings, imported metadata with save/reload, GET and terminal revision emission, refusal projections, private-token redaction, and the public-projection browser marker with timestamp drift.

Why It Matters

Regeneration can no longer erase a concurrent user turn, and a writable imported session can regenerate a completed WebUI-owned turn without authorizing an unowned foreign turn. Mid-stream restore also reuses the projected active prompt instead of rendering a duplicate when timestamps drift.

Verification

The public-projection browser regression fails on the prior head because the projected row is not selected, then passes after the marker fix. The focused regeneration, authority, browser, identity, and settlement tests pass with 35 tests. Python, JavaScript syntax, runtime ESLint, diff hygiene, and invariant coverage checks pass; the full suite and hosted browser validation remain CI-owned.

Risks / Follow-ups

Read-only, unmarked foreign, malformed-marker, fork-lineage, and runner-backed regeneration remain unavailable by design. Full CI and hosted browser validation remain the repository's CI checks; this change has no visible UI change and needs no screenshot section.

Contract Routing

Task type: bug fix, server-owned regeneration turn identity and transaction safety.

Touched areas: regeneration preview, locked start settlement, selected-turn ownership, GET and terminal revision consumers, browser pending-row reconciliation, and focused regression tests.

Relevant public docs: docs/CONTRACTS.md, docs/GUIDELINES.md, and the session SSE contract.

Scope boundaries: generic session operations, runner regeneration, browser request shape, schemas, session origin metadata, and public private-field redaction remain unchanged.

Evidence needed before claiming done: current hosted CI, browser smoke, and maintainer re-review must pass on the pushed head.

Model Used

GPT-5.6 via Codex CLI

[cutter-sh]

🎬 Cutter preview — PR #6677

Regenerate a chat response — Regeneration reuses the same user message and attachment instead of duplicating them.

[nesquena-hermes]

Thanks @rodboev — the regeneration identity, display-coordinate, gateway-settlement, cancel, and stale-recovery work is substantively strong, and the final no-token correction is valid. A focused exact-head warm-up found one containment gap before this can advance.

CORE: _active_turn_token escapes public message boundaries

The PR intentionally persists _active_turn_token as server-owned recovery authority. _public_session_messages() strips it on selected routes, but the shared redactor does not: api/helpers.py::redact_session_data() recursively redacts values without removing this key. As a result, /api/session/export serializes redact_session_data(s.__dict__) and exposes the token from both messages and context_messages. The authenticated share create/revoke responses and several direct message-bearing session responses also bypass the route-local projection. A sandboxed production-function probe reproduced the JSON-export leak at exact head 40b59f081d9f6e131f7d78bfd60db2ebabc4bcd3.

Fix specification

1. Put the removal in one shared, non-mutating public response projection (the shared redaction layer or a helper it always invokes), covering both messages and context_messages.
2. Route every outbound message-bearing boundary through it: primary GET, terminal/gateway payloads, JSON/HTML export inputs, share create/revoke, duplicate/update/truncate/branch/compression-recovery, and siblings.
3. Strip _active_turn_token from public JSON/session imports before persistence so callers cannot inject server-minted authority.
4. Add boundary tests proving the token remains on disk for recovery but is absent from all public responses/exports and imported/reloaded sessions.
5. Rebase the resulting patch onto current master; the present branch conflicts in api/routes.py.

Focused safe-wrapper evidence was otherwise strong: 235 passed, 2 skipped, 0 failed across regeneration, gateway, cancellation, run-journal, sidecar, and stale-pending recovery slices. The two skips are test-portability cases whose base-commit git show cannot access the outer worktree Git dir in the sandbox. This review is intentionally narrow: do not reopen the converged regeneration identity design.

[rodboev]

Thanks for catching this. The rework keeps the regeneration identity and display-coordinate design unchanged and closes the missing public containment boundary.

1. _active_turn_token now goes through the shared public message projection in api/helpers.py, covering both messages and context_messages without changing the stored session. GET, terminal and gateway payloads, exports, shares, mutation responses, compression responses, and the other message-bearing boundaries use that same rule.

2. JSON session import removes a caller-supplied token before the session is constructed or saved. Server-minted tokens remain persisted for settlement and restart recovery.

3. The api/routes.py conflict retained the display-space regeneration checks and current master’s branch and public-projection behavior. The route-local token stripper was removed.

The boundary tests cover exports, shares, local and gateway terminal responses, mutation responses, import and reload, runner and compression payloads, recovery after a partial assistant tail, strict coordinate validation, and the split between durable recovery state and public output.

[greptile-apps]

Greptile Summary

The PR makes regeneration admission and transcript replacement atomic under the per-session lock while adding durable ownership for imported and forked turns.

• Centralizes regeneration authority, revision validation, transcript truncation, and rollback.
• Projects private active-turn identity into a browser-safe marker for pending-row reconciliation.
• Aligns local and gateway settlement behavior and adds focused concurrency, ownership, projection, and browser tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
api/routes.py	Moves regeneration mutation, stream registration, persistence, and compensation into the locked start transaction.
api/session_ops.py	Adds canonical regeneration planning, revision hashing, selected-turn ownership checks, and aligned display/context truncation.
api/helpers.py	Redacts private turn identity while projecting the active user row through a public boolean marker.
api/streaming.py	Emits regeneration authority on terminal payloads and preserves retained active-turn identity during settlement.
api/gateway_chat.py	Reuses active-turn materialization and preserves regeneration context semantics in gateway settlement.
static/ui.js	Prefers the public active-turn marker when reconciling a pending prompt before existing identity and timestamp fallbacks.
static/messages.js	Routes browser regeneration through the revision-based chat-start contract.
static/sessions.js	Integrates the projected pending-turn identity into session transcript reconciliation.
tests/test_issue6611_regeneration_transaction.py	Covers locked settlement, rollback, and competing normal-send orderings.
tests/test_issue6611_regeneration_authority.py	Exercises revision issuance and canonical regeneration authority.
tests/test_issue6611_regenerate_turn_identity.py	Verifies durable ownership and rejection behavior across WebUI, imported, and forked turns.
tests/test_issue6611_regenerate_browser.py	Covers browser reconciliation using the projected active-turn marker despite timestamp drift.

Sequence Diagram

sequenceDiagram
    participant Browser
    participant Route as api/routes.py
    participant Authority as api/session_ops.py
    participant Session as Session persistence
    participant Worker as Streaming worker
    Browser->>Route: POST /api/chat/start (regenerate + revision)
    Route->>Authority: Plan regeneration
    Authority->>Authority: Reconcile transcript and validate ownership
    Route->>Route: Acquire session lock and revalidate revision
    Route->>Session: Truncate to retained user turn and stamp stream identity
    Route->>Worker: Start gated worker
    Route->>Session: Save accepted regeneration state
    Route->>Worker: Release worker
    Worker-->>Browser: Stream regenerated response

Loading

_{Reviews (15): Last reviewed commit: "ci: re-trigger (Playwright install hung ..." | Re-trigger Greptile}

[nesquena-hermes]

🔬 Gate certification — RED ⛔

Certified head: 6368e13f5bac18e002f454dd87c5391bc778d638
Rebased gate head: 3f84c33e94ebde51860329f9798a95cb58fbc980 on current origin/master d13e9221 (was 14 behind; overlap on routes.py/i18n.js/messages.js/ui.js auto-merged; rebased head sits directly on master).
Verdict: gate-fail — the token-containment half is correct, but the PR's own public-redaction of _active_turn_token breaks its own regeneration-row dedup: a refresh/reattach during regeneration materializes a DUPLICATE user row (reproduced first-party). A second reproduced regression: regeneration is rejected for sessions with string timestamps.

The direction is right and the token containment (the prior CR basis) is genuinely CLOSED — but the redaction and the regeneration-identity mechanism collide.

What I ran

Leg	Result
Threat scan (via local diff)	CLEAN (score 0)
Rebase-first	Clean rebase onto d13e9221 (14 behind; 4-file overlap auto-merged). Gated the rebased head.
Full pytest suite (sandboxed, -p no:xdist, to completion)	14,602 passed, 95 skipped, 1 xfailed, 2 xpassed, 34 subtests + 18 failed / 2 errors. The failing node set is byte-identical to the same-box frozen-master (d13e9221) control (established baseline + the master-batch-B test_glm_5_3_catalog sandbox artifact) → zero PR-owned suite failures. All PR-owned test files (test_issue6611_*, test_security_redaction, etc.) pass.
Codex (reproduce)	SHIP ONLY WITH FIXES — 2 CORE, both VERIFIED
Opus 4.8 (senior review)	run timed out / unavailable this pass (advisor infra hang past the 1200s cap; re-attempt in progress). Verdict rests on Codex + first-party reproduction.
Reviewer independent repro	Finding #1 reproduced first-party with the real static/ui.js getPendingSessionMessage/_pendingActiveTurnUserMessage

MUST-FIX (blocker, reproduced) — refresh during regeneration duplicates the user row

api/helpers.py:22 × static/ui.js:10673 (_pendingActiveTurnUserMessage). The PR correctly adds _active_turn_token to _PUBLIC_MESSAGE_INTERNAL_FIELDS so it's stripped from every public GET/export/share/gateway payload (the CR fix). But getPendingSessionMessage → _pendingActiveTurnUserMessage dedupes the current turn's user row by exactly two identity signals: (1) _activeTurnTokenMatches (the token), or (2) the row timestamp within a 1e-6 epsilon of pending_started_at. For a regenerated turn the retained user row keeps its original timestamp (from first send) while pending_started_at is the new regeneration start — so signal (2) can't match, leaving the token as the ONLY identity. Public redaction strips it → neither signal matches → the pending prompt is materialized as a SECOND row.

Reproduced first-party (extracted the real ui.js helpers, drove them with a redacted refresh payload):

refresh payload w/ token STRIPPED (public redaction) → getPendingSessionMessage MATERIALIZES a new _pending user row (DUPLICATE)
control: same payload w/ token PRESENT (disk shape)  → adopts the existing row (correct)

Codex independently verified the same via the real frontend merge (inserted: true, two identical user rows). This is the exact duplicate-user-row class #6611 is about, reachable on a normal (non-imported) session by a refresh/reattach during regeneration.

Fix spec (Codex + reviewer agree): project a non-secret active-row marker (e.g. a redaction-safe boolean/opaque id that is NOT the raw stream token) before stripping _active_turn_token, consume that marker in _pendingActiveTurnUserMessage() as the exact-identity signal for the redacted payload, and scrub any caller-supplied marker on import. This preserves both the security containment AND the regeneration dedup.

MUST-FIX (blocker, reproduced) — regeneration rejected for string-timestamp sessions

static/ui.js:19183 (typeof selectedTimestamp !== 'number') and api/routes.py:9266 (not isinstance(timestamp, (int, float))). Both new validators reject a regeneration target whose timestamp is a non-empty string, raising regen_stale_target / "Malformed regeneration target." The codebase's own _normalized_message_timestamp_for_key / _message_timestamp_as_float (models.py) explicitly handle string/ISO timestamps, so such rows exist in real data (imported CLI transcripts, numeric-string timestamps). Codex reproduced: an exact ISO-timestamp target returns stale_regeneration_target. This whole _regeneration_target_row validator is NEW in this PR (absent on master), so the restriction is PR-introduced.

Reachability note: most acute for imported sessions carrying string timestamps; if those are read-only they can't regenerate (lower blast radius), but the validator also rejects any legitimately non-numeric timestamp a real row could carry. Fix spec: accept non-empty string timestamps in BOTH validators and compare the original value exactly (the comparison downstream is ==, which works for strings), rather than type-gating on number/int|float.

Verified SOUND (token containment — the prior CR basis is CLOSED)

• _active_turn_token is in _PUBLIC_MESSAGE_INTERNAL_FIELDS (helpers.py:22) and stripped at all three redaction layers — _scrub_alias_record (:1006), _public_message_projection (:1126), redact_session_data (:1215) — over BOTH messages and context_messages (via _redact_messages→_public_message_projection). Retained on disk for recovery. (This containment is exactly why the dedup breaks — the fix must reconcile the two, not undo the containment.)

Recommendation to the next agent

Do not merge. Both are contributor-level fixes reconciling the security containment with the regeneration-identity mechanism: (1) a redaction-safe active-row marker for the frontend dedup, (2) accept string timestamps in both regeneration validators. changes-requested applied; re-gate at the new head. Preserve @rodboev attribution — the token containment is correct; the two collisions with the redaction just need closing.

_{Gate leg: threat scan → full suite w/ frozen-master baseline reconciliation → Codex reproduce → first-party reproduction of finding #1 + master-delta on finding #2. Opus 4.8 senior leg hit an advisor-infra hang this pass and is being re-attempted; the RED verdict is already established by two independent reproductions (Codex + reviewer) of a reachable regression, so it does not depend on the Opus leg.}

[rodboev]

Thanks, the gate found two separate identity regressions: redacting _active_turn_token can materialize a duplicate user row during refresh, and both regeneration validators reject legitimate non-empty string timestamps.

1. I’ll keep _active_turn_token private and change the shared projection in api/helpers.py to emit a server-derived _active_turn_user boolean only on the current token-owned user row. static/ui.js::_pendingActiveTurnUserMessage() will use that marker as exact identity after refresh, while retaining the internal-token and precision-only timestamp fallbacks for existing paths.
2. JSON import will scrub caller-supplied _active_turn_user values alongside _active_turn_token, so the marker can’t be forged into persisted session data. Opaque keys inside user content and tool arguments will remain untouched.
3. The validators in static/ui.js::regenerateResponse() and api/routes.py::_regeneration_target_row() will accept numeric and non-empty string timestamps, carry the original value unchanged, and keep exact comparisons. Empty strings, booleans, objects, stale values, and mixed string/number identities will still fail closed.
4. The shared display merge now receives active-turn provenance from gateway and synchronous settlement, so stable IDs remain attached to eager checkpoints. Truncation and state replay use a dedicated UTC-safe timestamp parser, while the generic message identity parser stays unchanged. Unsupported runner regeneration has its own 409 classification and frontend cleanup path.

I’ll preserve the existing cancellation, recovery, gateway, lineage, provider, and ordinary-send behavior; these changes stay within the active-row projection, regeneration validation, and shared truncation-replay helpers.

[nesquena-hermes]

🔬 Gate certification — RED ⛔ (re-gate)

Certified head: cd08f80ecf5abc9d1d2c65b4564695f723183572
Rebased gate head: 94ce46310dd7501c933c0a2eceb6c6241dbac5ea on current origin/master e88aff43 (was 2 behind; clean rebase).
Verdict: gate-fail — both findings from my prior gate-fail are genuinely CLOSED (Codex + Opus + reviewer all confirm the _active_turn_user marker fix + string-timestamp acceptance). But the re-push introduced a NEW CORE data-loss regression: on a runner-backed session, regenerate now permanently deletes the answer and then fails with a 409, because a new unconditional unsupported_regeneration_backend early-return runs AFTER the frontend has already truncated the transcript. Codex reproduced it; reviewer confirmed it against the diff + master.

Prior findings CLOSED (confirmed 3 ways)

1. Refresh-during-regeneration duplicate user row → CLOSED. The fix projects a NON-SECRET _active_turn_user marker in _public_message_projection (helpers.py:1120) BEFORE the secret _active_turn_token is stripped; the marker is set only when a row's token equals the token rebuilt from the session's CURRENT (active_stream_id, pending_started_at) (build_active_turn_token, process_event_utils.py:141). Frontend consumes it at ui.js:10721/10748. Stale rows are NOT marked (fresh uuid.uuid4().hex stream_id per turn ⇒ different token), and no marker is emitted when active_stream_id is None. Secret token still stripped from all public surfaces.
2. String/ISO-timestamp regeneration → CLOSED. Numeric-string and ISO timestamps are now accepted and parsed.

What I ran

Leg	Result
Threat scan	CLEAN
Rebase-first	Clean onto e88aff43 (2 behind)
Full pytest suite (sandboxed, to completion)	14,644 passed + 10 failed / 2 errors — all in the established frozen-master baseline → zero PR-owned suite failures; PR's own tests (test_issue6611_* ×3, test_security_redaction, test_session_public_share, run_journal, watermark, cancel-data-loss) all pass
Codex (reproduce)	SHIP ONLY WITH FIXES — prior blockers closed; 1 NEW CORE data-loss on runner backend, verified with an isolated truncate/start repro
Opus 4.8 (senior)	GREEN / APPROVE — confirmed prior findings closed + marker sound + no token leak + watermark is a fix; did NOT exercise the runner-backend regenerate path
Reviewer independent	Both prior findings confirmed closed; NEW regression confirmed against the diff (guard is PR-added, absent on master) + the frontend truncate-before-start ordering

MUST-FIX (NEW CORE — data loss on runner-backed regenerate)

api/routes.py:22329 + static/ui.js:19205. The PR ADDS (all + lines, absent on master) an unconditional early return in /api/chat/start:

if runtime_adapter_enabled() or runtime_adapter_runner_enabled():
    if regenerate_target is not None and runtime_adapter_runner_enabled():
        return {"error": "Regeneration is not supported by the runner backend.",
                "code": "unsupported_regeneration_backend", "_status": 409}

But the frontend regenerate flow (ui.js) does await api('/api/session/truncate', …) FIRST — permanently truncating the transcript (deleting the assistant answer) — and only THEN await send({regenerateTarget}) → /api/chat/start. On a runner-backed session the start returns the new 409; the catch sets a status but cannot restore the already-deleted answer, and the runner is never invoked. Net: a runner-backend user who clicks regenerate loses the answer AND gets an error, with no regeneration. Codex verified with an isolated truncate/start reproduction (persisted history = only the retained user prompt; runner start calls = 0); origin/master invoked the runner after truncation (no such guard existed). Fix (either): (a) remove the runner-only rejection at routes.py:22329–22334 so adapter.start_run executes the regenerate; or (b) if the runner genuinely can't regenerate, gate the FRONTEND truncate behind a backend-capability check so it never deletes the answer before a rejection (and/or restore on 409). Add a test proving a runner-backed regenerate either runs OR leaves the transcript intact on rejection.

Non-blocking (Opus advisories, do not gate)

• The all(ts is None) preserve branch could retain a deleted suffix only in a degenerate legacy(all-timestamp-less)+positive-watermark+no-sidecar state — contradictory in practice, and strictly less destructive than master's full wipe.
• _activeTurnTokenMatches (ui.js:10675) is now dead on public surfaces (raw token stripped); the marker + timestamp-epsilon fallback carry the logic. Harmless leftover.

Verified SOUND

• The marker mechanism is correct and secret-safe (Opus + Codex + reviewer): per-request active-turn scoping via a fresh-uuid stream_id, no over-match of stale rows, secret token stripped from messages/context_messages/share/export.
• String/ISO timestamp regeneration works; replay/watermark, gateway, cancellation, and sharing focused checks pass; watermark change is a net fix.

Recommendation to the next agent

Do not merge. The regeneration-identity fix I bounced for is genuinely resolved and the marker design is sound — but the re-push introduced a data-loss path on runner-backed sessions (delete-then-409). One targeted fix (remove the runner rejection so the runner regenerates, OR guard the frontend truncate behind a capability check). changes-requested applied; re-gate at the new head. Preserve @rodboev attribution — the marker work is exactly right; the runner-backend regenerate guard is the new blocker. (Codex/Opus split resolved by reproduction + diff evidence: the 409 guard is PR-added and absent on master, and the frontend truncates before it fires.)

_{Gate leg: threat scan → full suite w/ frozen-master baseline reconciliation → Codex reproduce (isolated truncate/start repro) → Opus 4.8 senior (GREEN on marker/redaction/watermark) → reviewer diff + ordering confirmation. Backend crown-jewel streaming/regeneration; no browser/UX leg (the regression is a backend guard + frontend ordering, confirmed by code + Codex's runtime repro).}

[rodboev]

I rebuilt this around a single server-owned start transaction. Regeneration now resolves the latest writable local exchange under the session lock, keeps the original user row, and releases a local or gateway worker only after the updated session is saved. Runner-backed regeneration returns before session lookup, workspace recovery, or transcript mutation, while preparation, registration, thread-start, and save failures restore the complete prior state.

Settlement also reuses the exact retained row in display and model context, including structured content, attachments, source, stable identity, and metadata. Error, cancel, reload, and state.db recovery therefore keep one user row without losing the replacement assistant.

[rodboev]

Fixed in 6c6d5c3a.

Regeneration stays a single server-owned operation. Runner-backed requests are refused before session lookup or mutation, the route now cleans stale persisted streams before selecting a turn, and workspace recovery waits until start acceptance before updating the sidecar. Regeneration also rejects external session and row sources, while legacy-journal, gateway, and persisted runner-refusal paths have direct coverage.

The CI regressions are fixed as well: ordinary gateway pause recovery keeps its prior context fallback, regeneration marks its authoritative empty context explicitly, active-turn checkpoint lookup fails closed without a token, and the affected static regression expectations now match the server-owned flow.

[nesquena-hermes]

🔬 Gate certification — RED ⛔ (re-gate — data-loss CLOSED, 2 new CORE from the rewrite)

Certified head: aeae998f28ab1312ab02ca3724ab5c827880fd44
Rebased gate head: f481f84dd1e674c5257144c12d1fecc863293781 on current origin/master 69d5647b (1 behind; clean rebase).
Verdict: gate-fail — the cycle-3 data-loss (truncate-then-409 on runner backends) is genuinely CLOSED, and the new regeneration_revision atomicity token is race-safe (Codex + Opus + reviewer all confirm — no stale-truncate, no double-fire, no TOCTOU). But the "make regeneration start atomic" rewrite introduced TWO new CORE regressions Codex reproduced: (1) the regenerate button silently dies after every normal turn, and (2) on a state.db-recovered session the revision hashes/selects the wrong (sidecar-only) transcript and can delete recovered history. Both need fixing.

CLOSED — cycle-3 data-loss + atomicity (confirmed 3 ways)

• Cycle-3 data-loss CLOSED: startRegeneration (messages.js:1956) snapshots S.messages, truncates CLIENT-SIDE only, calls /api/chat/start with a regeneration_revision token, and RESTORES the snapshot on ANY error. Codex: "runner-backend 409 restores the browser snapshot without server mutation." No server truncate-before-commit.
• regeneration_revision race-safe (Opus reproduced hard; reviewer confirmed): it's a SHA-256 content hash over (session_id, messages, context_messages, watermark, boundary); resolve_regeneration_turn re-checks it under _get_session_agent_lock and apply_regeneration_truncation re-checks revision + row_digest + message_count again before truncating — no lock release between (no TOCTOU). Stale-revision → False (session unmutated); double-fire → exactly one winner (winner bumps the hash); retry after reject works.

What I ran

Leg	Result
Threat scan	CLEAN
Rebase-first	Clean onto 69d5647b (1 behind)
Full pytest suite (sandboxed, to completion)	14,642 passed + 9 failed / 2 errors — all in the established frozen-master baseline → zero PR-owned suite failures; PR's own regenerate tests pass
Codex (reproduce)	SHIP ONLY WITH FIXES — 2 CORE (regenerate-dead-after-turn; recovered-history wrong-prompt); confirmed data-loss closed + atomicity safe
Opus 4.8 (senior atomicity)	GREEN / APPROVE — verified data-loss closed + revision race-safe; did NOT examine the terminal-payload or state.db-reconciliation paths
Reviewer independent	Data-loss closed + atomicity sound (read); finding #1 confirmed structurally (revision is GET-only; done handler wholesale-replaces S.session)

MUST-FIX #1 (CORE, reproduced) — regenerate silently dies after every normal turn

static/ui.js:19236 + api/streaming.py (terminal payload) + static/messages.js:6188. regeneration_revision is added to the session payload only at the GET /api/session path (routes.py:13167) — it appears nowhere in streaming.py, so the terminal done/error/cancel SSE payloads (_session_payload_with_full_messages) omit it. The done handler does S.session = d.session (messages.js:6188, wholesale replace), so after any turn S.session.regeneration_revision is undefined → regenerateResponse's guard if(!S.session || S.busy || !S.session.regeneration_revision) return (ui.js:19236) silently bails. Codex verified start_calls=0; reviewer confirmed the field is GET-only + the done handler replaces wholesale. Impact: the regenerate button does nothing after every normal turn until a full session reload. Fix: include a fresh regeneration_revision in every writable terminal session payload (local + gateway done, error, cancel) before it replaces S.session.

MUST-FIX #2 (CORE, Codex-reproduced) — wrong-prompt selection + recovered-history data-loss on state.db-reconciled sessions

api/routes.py:13167 + api/session_ops.py:86. The GET displays the transcript reconciled with state.db (routes.py:12925), but regeneration_revision hashes only sidecar s.messages, and resolve_regeneration_turn selects the turn only from s.messages. Codex verified with a 2-row sidecar + 4-row recovered transcript: the user clicked "latest recovered prompt," but the accepted revision selected the "older prompt," and settlement permanently reduced the transcript to the older prompt + its replacement — recovered history deleted, wrong turn regenerated. This is the same data-integrity class this PR keeps fighting, now on the recovered-session path. Fix: generate, validate, select, and truncate against the SAME fully-reconciled transcript (the one the GET displays) under the session lock — not sidecar s.messages alone.

Verified SOUND (all three legs)

• Cycle-3 runner-backend 409 data-loss closed (client restores, no server mutation).
• regeneration_revision atomicity race-safe (stale/double-fire/TOCTOU all fail-closed; Opus 5-probe sweep + reviewer read).
• Display/context truncate parity (truncate_session_at_keep/truncate_context_for_display_keep, same helpers as /api/session/truncate); watermark/boundary set; crash-recovery merge collapses to [prompt, new-answer] (no resurrected answer).
• Earlier fixes intact: _active_turn_user dedup + numeric-string/ISO timestamps (Codex).

Recommendation to the next agent

Do not merge. The hard part — the cycle-3 data-loss and the atomicity token — is genuinely done and race-safe. Two fixes remain, both concrete: (1) emit regeneration_revision on every terminal session payload (not just GET) so regenerate survives a normal turn; (2) key the revision + turn-selection + truncation off the state.db-reconciled transcript, not sidecar s.messages, so a recovered session can't lose history. changes-requested applied; re-gate at the new head. Preserve @rodboev attribution — the atomic rewrite is the right architecture; these are two reachability gaps in wiring it to the terminal-payload and recovered-transcript paths. (Codex/Opus split resolved by reproduction: Opus GREEN verified the atomicity question my brief emphasized but didn't cover the two paths Codex + reviewer reproduced.)

_{Gate leg: threat scan → full suite w/ frozen-master baseline reconciliation → Codex reproduce (2 CORE, data-loss+atomicity confirmed closed) → Opus 4.8 senior atomicity (GREEN, race-safe) → first-party reproduction of finding #1 (revision GET-only + wholesale done-replace) + read of the atomicity path. Mixed backend + visible regenerate; the two blockers are backend/wiring, confirmed by code + Codex's runtime repro.}

[rodboev]

Thanks, I rebuilt the flow around one canonical transcript instead of adding another turn-specific guard.

The server now reads one state.db-reconciled snapshot for the terminal payload, GET authority, revision, turn selection, and truncation. Terminal done, error, and cancel payloads carry the same rows they hash, and the browser waits for the full transcript before requiring the revision. Parent-only, partial, foreign, stale, active, pending, and trailing-tool states fail closed.

The start path snapshots before canonical assignment and restores that snapshot on stale truncation or any pre-acceptance failure. Retained timestamps and complete attachment metadata survive regeneration.

The focused issue and adjacent tests pass locally. The hosted checks will provide the served browser, runner-refusal, and compensation evidence after this head is pushed.

[nesquena-hermes]

Warm-up re-gate — changes requested

Reviewed head: 20716c21db722ddacee7ca6e6f5a4b96d5a25b82

Thanks for rebuilding this around one canonical transcript. The two blockers from the prior gate are closed: terminal payloads now carry a fresh regeneration revision, and revision/turn selection use one state.db-reconciled transcript. The private _active_turn_token boundary also remains closed.

A focused exact-head re-gate found two deterministic state-integrity blockers in the new transaction:

1. The authorized canonical context is discarded before truncation

api/routes.py::_start_regeneration_stream_locked() computes and validates both canonical_rows and canonical_context (21697-21707), but installs only canonical_rows on the session (21708-21710). apply_regeneration_truncation() then truncates session.context_messages, which can still be the stale sidecar context (api/session_ops.py:284-320).

On a state.db-recovered session where canonical context contains rows absent from the sidecar, the revision authorizes the recovered pair but the accepted mutation persists a prefix derived from stale context. That can omit recovered model history from the regenerated turn.

Fix: install both authorized values before mutation:

s.messages = copy.deepcopy(canonical_rows)
s.context_messages = copy.deepcopy(canonical_context)

Alternatively, pass the immutable pair into one apply helper that verifies and truncates that exact pair without another authority read. Add a production-composed regression where canonical state.db context differs from sidecar context and prove the recovered prefix survives local and gateway settlement.

2. Rejected starts no longer roll back exactly

api/session_ops.py::snapshot_regeneration_state() snapshots a fixed field allowlist (53-68), but this same start path mutates fields outside it:

• _prepare_chat_start_session_for_stream() clears post_compression_context_tokens_estimate.
• Session.save() changes updated_at and _anchor_scene_index before an I/O failure can escape.
• set_last_workspace(workspace) runs before the accepting s.save() and is not compensated.

The exception path restores only the allowlist and can persist that incomplete restoration with touch_updated_at=False (api/routes.py:21838-21855). A failed pre-acceptance start can therefore return failure while leaving session/global state changed. The previous non-ancestor revision used a complete session.__dict__ snapshot and had a failure-stage matrix; both protections were removed in this rewrite.

Fix: restore a complete transaction snapshot (prefer a deep copy of the mutable session state, or a mechanically complete transaction object), and move set_last_workspace() until after successful s.save() but before worker release. Restore the prepare/journal/registration/thread/save failure matrix and assert exact in-memory plus persisted rollback, including non-null post-compression state and the original timestamps/fingerprints.

Gate evidence

Threat scan: CLEAN. Two mandatory bubblewrap-sandboxed targeted slices passed: 20 passed for the issue-6611 files and 14 passed across existing settlement, terminal-payload, redaction, and share coverage. Those tests do not compose either failing path above; the deleted failure/recovered-context tests are the gap.

No merge or contributor-branch write was performed.

[rodboev]

Thanks for the exact re-gate. I rebuilt the regeneration start around one lock-scoped display/context pair and a complete session snapshot. The accepted save is now the boundary before workspace updates and worker release, stale plans leave newer accepted state untouched, and failed preparation restores the full pre-acceptance session. The canonical context row receives the same private turn marker as the retained display row, including recovered rows without metadata, and fork-child markers are bound to the current child session.

The issue artifact, recovered-context, rollback, lineage, settlement, and adjacent-operation checks pass in the focused suite.

[rodboev]

Thanks, the production blockers are closed. I kept that implementation unchanged and fixed the two CI integration failures.

1. The extracted newSession() harness now loads the real _adoptRegenerationRevision() helper before it evaluates and executes newSession(), so the explicit empty-composer override assertion still covers the complete production path.

2. The issue-6611 browser and transaction tests now share a minimal tracked fixture under tests/fixtures. They no longer read .claude/pr-sweep or any file outside a clean checkout.

3. The regeneration revision adoption calls remain active in production and are guarded for the repository's isolated newSession() and loadSession() harnesses, preserving the existing session-rotation and unread-visit contracts.

I also corrected the PR verification text to describe the tracked fixture and the current focused coverage.

[nesquena-hermes]

Deep re-gate: test-portability findings are resolved — two CORE data-integrity regressions remain

Thanks @rodboev — the round-3 test-portability blockers are both closed: the issue-6611 reproduction is now self-contained (tests/fixtures/issue6611_regeneration_rows.json + tests/_issue6611_fixture.py, no more .claude/pr-sweep dependency) and the extracted newSession() harness contract is preserved. The atomic-plan/settlement architecture (single authority read in plan_regeneration, re-validation before mutation in apply_regeneration_plan, full-__dict__ snapshot/restore) is sound in the single-caller case.

Deep review (adversarial reproduction) surfaced two CORE regressions that the green suite does not cover:

1. [CORE] Concurrent send is erased by the request-time rollback — api/routes.py:23393

regeneration_snapshot = snapshot_regeneration_state(s) is captured at request time, before the session LOCK, and _restore_regeneration_preacceptance() restores that whole Session.__dict__. If a concurrent send acquires the session between the snapshot and the restore (setting active_stream_id + pending_user_message), a regeneration that then loses revision validation restores the stale snapshot and wipes the concurrent winner.

Reproduced: regeneration returned 409 while winner-stream and its pending prompt were reset to None — a silent loss of the concurrent turn.

Fix: remove the request-time whole-session rollback. Make the locked regeneration settlement the sole rollback owner — snapshot and restore only inside the per-session lock, after revision revalidation. Keep the pre-lock validation side-effect-free, or revert only transaction-owned fields via compare-and-swap. Add a concurrent-winner reproduction (regeneration loses the revision race → the concurrent send's stream + pending prompt survive intact).

2. [CORE] Regeneration disabled for all writable imported sessions — api/session_ops.py:216

The blanket is_cli_session / foreign-source rejection at lines 216, 277, 287 disables regeneration for every writable imported CLI/TUI/Desktop session — including a completed WebUI-owned turn in such a session. Those sessions remain sendable, so regeneration becoming unavailable is a new functionality loss, not a safety guard.

Fix: replace the blanket rejection with selected-turn ownership validation — permit regeneration of writable imported sessions only when the final user row carries a durable WebUI turn marker (e.g. _active_turn_token), while continuing to reject unmarked foreign turns. This keeps the foreign-turn source guard intact without denying provably WebUI-owned turns.

Both findings independently reproduced at head da6a5cb9. The direction is right and the atomicity core is solid — these are the last two before this critical-capability fix can land. Happy to re-gate immediately once the rollback is lock-scoped and the imported-session gate is ownership-based.

[rodboev]

Thanks for the deep re-gate. The two remaining data-integrity cases are fixed in e3224a00:

• Regeneration preview no longer refreshes imported messages or owns a request-time snapshot. Rollback stays inside the session-locked transaction, so a concurrent send that wins the lock remains intact.
• Imported-session eligibility now follows the selected final user row. A valid durable WebUI turn token permits regeneration; read-only, unmarked foreign, malformed, and invalid fork rows remain rejected.

The added coverage exercises both lock orderings, save/reload, GET and terminal revision consumers, and public token redaction.

[nesquena-hermes]

Deep re-gate: both data-integrity CORE findings are CLOSED — but a new server→client marker mismatch double-renders the active prompt

Excellent progress @rodboev — both cycle-6 blockers are verified closed (reproduced in real Chromium + memory):

• Concurrent-send erasure (finding #1): CLOSED. The concurrent-send winner now survives the losing regeneration's 409, both in memory and after reload. The whole-__dict__ rollback remains, but the snapshot and every restore now occur inside the session lock after revision revalidation, so no concurrent writer can interleave. lock_held=True correctly uses nullcontext() — no re-acquire of the non-reentrant lock (avoids the #6955/#7107 deadlock class), and the worker is gated only through start settlement, not the streaming lifetime.
• Imported-turn regeneration loss (finding #2): CLOSED. Completed WebUI-owned imported turns are accepted; unmarked, malformed, earlier-only, read-only, and foreign final turns remain correctly rejected — precise, not blanket.
• Stale-revision rejection, plan/apply TOCTOU protection, atomic display/context installation all intact. The race test is substantive. 51 regeneration + 41 neighboring tests pass.

Must fix: the new _active_turn_user marker isn't recognized client-side → duplicate pending prompt on mid-stream restore — static/ui.js:10799

The backend now replaces the raw _active_turn_token with a boolean _active_turn_user: true in the public projection (api/helpers.py:1144) — good, it doesn't leak the token. But _pendingActiveTurnUserMessage() (ui.js:10800) still checks only the raw token:

if(!msg||typeof msg._active_turn_token!=='string') return false;

For a public-projected message (token stripped, _active_turn_user:true set), this early-returns false → the active-turn identity isn't recognized → mid-stream restore/reattach falls through and materializes another pending user row when timestamps drift. Reproduced: the active user prompt renders twice.

Fix: in _pendingActiveTurnUserMessage, treat msg._active_turn_user === true as authoritative active-turn identity before the token/timestamp fallback, retaining raw-token compatibility for the non-projected path. Add a backend-public-projection→browser probe proving a drifted timestamp does not create a duplicate pending prompt.

This is a clean cross-layer contract mismatch — the server-side projection changed the marker shape and the client reconciliation wasn't updated to read it. Independently confirmed at the code level (helpers.py:1144 sets the boolean; ui.js:10800 only reads the string token). Everything else is solid — this is the last gap. Re-request review once the client recognizes the marker.

[rodboev]

Thanks for catching the projection boundary mismatch. The server correctly replaces the private token with _active_turn_user: true, but the browser helper was still recognizing only the private representation.

I’ll make _active_turn_user === true authoritative in _pendingActiveTurnUserMessage, while keeping the raw-token and precision timestamp paths for private or partially projected transcripts. I’ll also add a cross-layer regression that creates the real public projection, confirms the private token is absent, feeds the projected row to the browser helper with timestamp drift, and verifies that the existing prompt row is reused.

Resolved — 5th gate at this head is SAFE TO SHIP (Codex + full suite + independent verify; both CORE data-integrity findings closed, client-marker residual fixed, no regression).

[nesquena-hermes]

Shipped in exp-v0.52.248 — thanks @rodboev. This landed after a thorough multi-round convergence: both core data-integrity guarantees are now in place (a concurrent send during regeneration survives instead of being rolled back, and completed imported CLI/TUI/desktop turns can be regenerated while malformed/read-only/foreign finals stay rejected), plus the active-turn marker is carried through the public projection so a mid-stream reload renders the pending prompt once. Gate: Codex SAFE + full suite green + independent code-level verification of the snapshot/restore-under-lock and marker spoof-safety. Appreciate the persistence through the review rounds.