Skip to content

fix(agent): persist_session mutates copy, not live messages (#48677) - #48817

Closed
kyssta-exe wants to merge 1 commit into
NousResearch:mainfrom
kyssta-exe:auto-fix/issue-48677
Closed

fix(agent): persist_session mutates copy, not live messages (#48677)#48817
kyssta-exe wants to merge 1 commit into
NousResearch:mainfrom
kyssta-exe:auto-fix/issue-48677

Conversation

@kyssta-exe

Copy link
Copy Markdown
Contributor

Fixes #48677. The early-crash-resilience persist in build_turn_context called _apply_persist_user_message_override on the live messages list, stripping observed group-chat context before the API call was built. Now _persist_session works on a shallow copy so the override only affects persistence, not the API request.

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists labels Jun 19, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #48694 — both fix #48677 (early crash-resilience persist mutating the live messages list and stripping observed group-chat context before api_messages is built). Same approach: persist on a copy of the messages so _apply_persist_user_message_override no longer touches the API-facing list. #48694 is the earlier open PR and the more complete version (adds a _messages_for_persistence helper that also preserves SQLite identity-dedup and skips multimodal list-content blocks). Consolidating there.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jun 21, 2026
@teknium1

teknium1 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the careful diagnosis here, Kyssta — the writeup on #48677 was genuinely precise (the early crash-resilience persist mutating the shared messages list in place before api_messages is built is exactly right, and the string-vs-list guard gap is real).

We're closing this rather than salvaging it, though. The fix as written also conflicts with the current _flush_messages_to_session_db dedup, which post-dates this PR: it tracks already-persisted messages by dict identity (id(msg) in history_ids) and stamps a _DB_PERSISTED_MARKER on each live dict. Passing a shallow copy breaks both — history messages fail the identity check and get re-written every turn (resurrecting the duplicate-write bug #860), and the marker lands on the throwaway copy so nothing is ever deduped again.

More fundamentally, we don't want to patch the observed-context delivery path here at all. It's built on the persist-override mechanism (send-wrapped-to-API, save-bare), and that mechanism is fighting itself in this seam. We'd rather rethink that delivery than bolt copy-on-persist onto it. Closing both this and #48677.

Your authorship stays in the record via this PR. Appreciate the work.

@teknium1 teknium1 closed this Jul 1, 2026
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Jul 1, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
kshitijk4poor pushed a commit that referenced this pull request Jul 1, 2026
… list (#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in #56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from #48817; chokepoint reworked to coexist with the marker-based
dedup (#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
… list (NousResearch#48677)

The persist user-message override was applied in place to the live messages
list. On the early crash-resilience persist (which runs BEFORE api_messages is
built), that stripped observed group-chat context off the live user message and
silently dropped it when observe_unmentioned_group_messages was enabled.

Fix at the single chokepoint: _flush_messages_to_session_db resolves the
override (idx/content/timestamp) locally and applies it ONLY to the row written
to the DB — the live dict is never mutated, so EVERY persist caller (early
persist, mid tool-loop flush, /resume, /branch) is protected uniformly. This
supersedes the earlier shallow-copy approach, which broke the intrinsic
_DB_PERSISTED_MARKER idempotency (copies never propagated the marker back to
the live dicts → duplicate rows) and closes the sibling class tracked in NousResearch#56303.

Trailing empty-response scaffolding is still dropped from the live list in
_persist_session (unchanged behavior).

Salvaged from NousResearch#48817; chokepoint reworked to coexist with the marker-based
dedup (NousResearch#50372).

Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P1 High — major feature broken, no workaround sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

observe_unmentioned_group_messages: observed context silently dropped before the API call (early persist mutates the shared messages list)

3 participants