Skip to content

fix(compression): fence durable transcript revisions - #72806

Open
ruizanthony wants to merge 5 commits into
NousResearch:mainfrom
ruizanthony:fix/compression-durable-transcript-revision
Open

fix(compression): fence durable transcript revisions#72806
ruizanthony wants to merge 5 commits into
NousResearch:mainfrom
ruizanthony:fix/compression-durable-transcript-revision

Conversation

@ruizanthony

@ruizanthony ruizanthony commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Fence compression against an atomic durable transcript revision instead of comparing a model-facing projection length with SQLite's active-row count.

This fixes the cross-repository race where WebUI can legitimately project fewer messages than the Agent's durable transcript, while a real concurrent mutation must still prevent stale compression output from publishing.

Root cause

len(durable_parent) > len(messages) was treated as evidence of concurrent writes. That is not a valid invariant across WebUI sanitization, deduplication, replay and sidecar projection. It caused false retries and could exhaust the provider context window. Conversely, without a durable revision/CAS boundary, a true concurrent rewrite could be missed between load and publication.

The previous PR head also introduced a separate CI regression: importing durable state through gateway.config -> gateway.session -> agent.turn_context -> agent.conversation_compression froze state.db before runtime HERMES_HOME resolution. The current head keeps those imports lazy; the existing runtime-home regression is green again.

Changes

  • capture messages and their durable revision from the same SQLite rows;
  • include active-row count, max active id, content digest and api_content sidecar digest;
  • propagate revision state through TurnContext/TurnRunner and refresh it after clean Agent writes;
  • re-check after acquiring the compression lease and at publication/mutator boundaries;
  • fail closed on stale or unverifiable revisions without replaying tools;
  • preserve current-turn partial output under explicit stale classification;
  • retain opt-in durable row IDs and coherent rewind/rotation snapshots;
  • keep durable-state imports lazy so runtime profile/home selection remains authoritative.

Dependency and pairing

Merge #71486 first, then rebase this PR onto the resulting main to drop the stacked duplicate and revalidate the new exact head together with WebUI.

Validation

  • critical Agent gate: 96 passed, 0 failed;
  • exact cross-repo WebUI↔Agent gate: 54 passed, 0 failed;
  • broader Agent matrix: 792 passed; 2 TUI config tests fail only because the local control environment lacks ruamel, reproduced identically on unmodified base (515 passed, same 2 failed);
  • Ruff on changed lazy-import files, py_compile, and git diff --check: passed.

No deployment or merge is performed by this PR update.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery area/compression Context compression and continuation sessions area/sessions Session lifecycle, resume, persistence, history sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state P2 Medium — degraded but workaround exists labels Jul 27, 2026
@ruizanthony
ruizanthony force-pushed the fix/compression-durable-transcript-revision branch 3 times, most recently from 72001f0 to 57cfe13 Compare July 29, 2026 18:21
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Rebased as a stacked follow-up on #71486, preserving both live-tip recovery and durable transcript revision fencing. Exact head: 57cfe13eb85ff7971c2b2fe6926ea50c4e2a813b. Combined targeted gate: 297 passed; diff check clean. Merge #71486 first, then this branch can be rebased once more onto upstream main to drop the stacked commit.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for tracing this to the durable-row-count assumption. The premise is confirmed on current main: agent/conversation_compression.py:1725-1744 still adopts durable history when len(durable_parent) > len(messages), which is not a valid concurrency test for sanitized or deduplicated projections.

Problems

  • The submitted branch is currently conflicting, and its base predates major changes to the affected state and gateway paths. Current hermes_state.py:6332-6386 includes the independent include_row_ids contract on get_messages_as_conversation, so the revision-aware load must be integrated without regressing that API surface.

Suggested changes

  • Salvage the revision/CAS design into the current state and gateway flow, retaining the PR's projection-mismatch, sidecar-mutation, and stale-publication tests against the current interfaces.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
@ruizanthony
ruizanthony force-pushed the fix/compression-durable-transcript-revision branch 2 times, most recently from 308b23e to 83a2fbf Compare August 3, 2026 10:08
@ruizanthony
ruizanthony force-pushed the fix/compression-durable-transcript-revision branch from 83a2fbf to 1bdac5d Compare August 3, 2026 13:02
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Rebased and repaired on current main. Exact review target: 1bdac5d4547914d70c134f06af0435568a35159c, stacked on #71486 exact head 5d4539dcfc782953fbc27bd630b96b245c7c7621. The prior CI failure is fixed by keeping durable-state imports lazy; 96/96 critical tests and 54/54 paired WebUI tests pass. Please review this exact SHA; after #71486 merges, this PR will be rebased once more and the paired gate rerun.

@ruizanthony

Copy link
Copy Markdown
Contributor Author

Pairing metadata correction: the WebUI tree is unchanged, but its final commit was republished with the required verified author/committer identity. Exact paired review target is now WebUI 0f7c4cef3f38aab591ae6fa6fcfdb76a869c669a with this Agent head 1bdac5d4547914d70c134f06af0435568a35159c; the exact pair remains 54/54 green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants