fix(matrix): preserve genuine thread replies and fallback chains by iainlane · Pull Request #125648 · NousResearch/hermes-agent

iainlane · 2026-09-27T19:57:52Z

Stack

All PRs in this section target main. This list records logical merge order. Nesting means the parent PR must merge before its child. Every prerequisite for a dependent must merge first. Only adjacent layers are shown; each linked PR description lists its other dependencies.

feat(matrix): implement spec-correct threading and reply semantics #62088
- fix(matrix): preserve genuine thread replies and fallback chains #125648

Summary

A Matrix threaded response can be a genuine reply to one event, or a continuation that includes a compatibility fallback for clients without thread support. Hermes marked every outbound thread relation as a fallback. Explicit replies kept their target but had the wrong flag, and plain continuations repeated the thread root for later text chunks and media sends.

Build the relation from the intended target. The first message of a response gets an m.thread relation whose m.in_reply_to points at its reply_to event, with is_falling_back: false. These sends are continuations:

later chunks and later messages of the same response;
a send anchored on the thread root, which in an auto-threaded room is the triggering event;
a send anchored on a message that Hermes sent in the thread, as the stream consumer does when it chains streamed chunks.

A continuation uses is_falling_back: true and points at the latest event known in that room and thread, including a chunk that Hermes delivered earlier.

The adapter keeps a bounded record for each room and thread, for up to 500 threads. The record contains the latest event and two lists of up to 32 recent events: the events that Hermes gave a genuine reply to, and the events that Hermes sent. Because of these lists, a busy acknowledgment, a second concurrent response or another user's post in the same thread does not make the rest of a response quote its trigger again. The adapter updates the record for inbound events and for successfully sent events only, so a failed send does not advance the chain. The matrix_thread_fallback_event_id send metadata overrides the fallback target; the thread-creation layer (#126341) uses it for the first reply in a new thread.

Other platforms

None. The relation decision is in the Matrix plugin. send() and the stream consumer are unchanged, and no gateway core file changes.

Configuration and documentation

No configuration or documentation changes.

Relationship to existing work

This salvages #42345 by @mijanx. Jan remains the author of the genuine-reply implementation commit (a849770a0c). The following commits add the latest-known fallback chain, the per-thread record of answered and sent events, and the client tests.

Recorded verification

The test results below were recorded at the historical revisions described here. They have not been rerun for this history and description cleanup.

All runs used scripts/run_tests.sh with mautrix 0.21.1 importable and no retries.

Unit tests cover explicit replies, replies anchored on the thread root, split final responses with a reply anchor, buffered and live streamed responses, a busy acknowledgment between two messages of a response, two interleaved responses, an inbound post between two streamed chunks, plain continuations, image and audio sends, and encrypted and unencrypted media payloads. The root, split and streamed cases failed before the continuation rule (4 failures), and the three interleaving cases failed before the per-thread lists replaced single values.
At 717506b6cb, 57 files pass 1066 tests with 1 skipped: every file that constructs the Matrix adapter, all stream-consumer files, the reply-injection, text-merge, Telegram topic, PII and session files, and the files that feat(matrix): implement spec-correct threading and reply semantics #62088 and feat(matrix): room naming and state-change notes #51804 touch, including the context-compressor and message-metadata tests.
A separate matrix-nio client sends two turns in one real Synapse thread and checks that each response is a genuine reply to the latest inbound event. A third case limits the gateway to 500-character messages and checks that only the first chunk of a split threaded answer is a genuine reply, and that each later chunk is a fallback to the previous chunk. That case failed before the change, when the second chunk also quoted the question. test_thread_context.py passed 3 of 3 cases at 717506b6cb in 447.9 seconds, including the gateway image build. Each client exchange has a 15-second deadline.
Ruff and git diff --check pass.

Known limitations

The live tests have no auto-thread case, because the live room has two members and auto-threading applies only to group rooms. A unit test covers a send anchored on the thread root.
Streamed, interleaved, encrypted and media relations are covered by adapter tests only.
Every successful threaded send updates the latest event of its thread, including status notices such as a busy acknowledgment. A later continuation's fallback can therefore point at a status notice.

Previous public source verification

The previous seven contributions were reconstructed above the exact published #62088 head 0fced0053ccad576a5d31f7cd8257cde734925d8. That publication retained all 65 prerequisite objects and signatures. The seven source authors, original author dates and complete messages are preserved. New committer and signature dates recorded that earlier delivery.

On source tree b5ab07a2ea087ba087b994db39a4a5f16cd30df5, the canonical runner passed 325 cases across 20 files with zero skips and strict warnings, using the actual mautrix SDK. The two selected regression cases failed before the correction. Ruff passes. Paired type checks report zero introduced findings; four inherited formatting failures remain. No new formatting debt belongs to these seven contributions. The earlier outer controllers failed their evidence guards, and their saved completion and source-byte qualifications were reviewed separately. Native Matrix validation remains separate.

This PR targets main. Merge #62088 and its listed prerequisites first. The captured source base is af90026aa09949579bd423d24def3d38f743cde0; advancing main alone does not require a new rebase or test run.

Previous published head: 0b6023d7212bec2abe8824589def56dd6eea921d. Review the previous seven contributions against their exact published prerequisite.

Earlier frozen-main prerequisite repair

The recorded branch reused the exact published #62088 prerequisite at e2bea25221b47fd8504d776962359d4be266c350. That source preserves both GatewayPluginInjectionMixin and prepend_turn_context_note at the upstream inbound import overlap and includes the accepted #125438 runner repair above frozen upstream 99a45ecc17cc51c0488ec3b8ccf7eaf483f7203d. Its 65 signed contributions are reused without signing them again. Only this layer's seven original contributions are replayed and signed with the configured key; their complete author and committer headers, dates, time zones, encodings, messages and trailers remain unchanged.

Recorded signed head: 2cf079c4fd36a83631e64d1e25d091ef3e8bfe27. Source tree: 0667929c2e4aa7956f1bf0b19e870c83e027ec49. The complete diff of these seven contributions is byte-identical to the previous public layer. Against the saved automatic merge with frozen upstream, only the inherited inbound import resolution, test runner and existing shell regression differ. All three paths match the exact published prerequisite byte for byte, so the prerequisite's two causal import controls apply to those unchanged source paths. Product code, SDK decoding, policy, privacy and fixtures retain the automatic merge result.

Seven complete files passed through the canonical runner on this exact composition: plugin-message injection (15), plugin-origin injection (3), Matrix adapter (210), session (67), shell contract (2), pending-fix strict helper (7) and E2E strict helper (19), 323 cases in total. The run uses two workers, the saved SDK addon, strict runtime, thread and unraisable-exception warnings, and no retries. No cases fail or skip. Whole-tree Ruff lint, shell syntax and repair whitespace checks pass.

Scoped type checks remain non-zero with exactly the prerequisite's 145 complete diagnostic records. The two checked inherited files also retain the prerequisite's formatting qualification; their source bytes are unchanged. Earlier own-layer, combined, static and native receipts below retain their recorded revisions and scopes. This refresh does not establish current native Matrix exchange or hosted-CI acceptance, and the PR remains a draft.

Review the recorded seven-contribution layer against the exact published prerequisite.

Earlier fixed-main alignment (recorded revision)

The source tree is dcd9ee5c6904103d11d33fc2703b95c1fbcd8fdf. That recorded refresh replayed the corrected seven public contributions above the replayed 65-contribution #62088 prefix onto the fixed main commit 33c9b1d7796b34d76e6b71fc4251a7197f7ffd80. Every recorded author and committer header, date, timezone and complete message is preserved. The existing message qualifiers and contributor trailers remain intact. The rewritten commits receive fresh signatures.

The gateway import conflict is resolved in the contribution that introduces the generic turn-context hook. Both the hook and the upstream media extraction remain available on the inbound path. The upstream crypto and media modules are unchanged. No additional behaviour is introduced.

The combined #62088 and #125648 source passed 409 cases across 24 files through the canonical runner, using the saved mautrix 0.21.1 SDK, with zero failures, skips or retries and strict runtime, thread and unraisable-exception warnings. Two causal controls each removed one import from the gateway conflict resolution; the existing production contract failed at the corresponding missing import. Restoring both imports passed that contract in the combined gate.

Ruff checks and complete source diff checks pass. Paired type checks contain the same 205 inherited diagnostics, with no added findings. Formatting remains qualified: 40 files require formatting, and every additional formatting difference is inherited from the fixed main source. The first harness attempt failed before collection because an older receipt plugin required a bytecode flag that the canonical environment scrub removed. The owned pytest plugin correction retained that assertion; all failed receipts remain available. This refresh does not establish a current native Matrix exchange, hosted checks or a repository-wide type or formatting pass. The preceding 201-case and 325-case source results remain historical baselines; the recorded combined gate applies to the combined source at that revision.

Review the recorded seven contributions against their recorded #62088 prefix.

Current bounded prerequisite composition

This revision uses the exact published #62088 prerequisite at f679a1acb683fbed3cf91dbaea36ca7cb5d9b10a, which incorporates the concrete compatibility repair against upstream 302c5d9f679333ae957c256d76a3dff35120f4e0. All 65 signed prerequisite objects are reused unchanged. The logical Stack above remains unchanged. Only this layer's seven current contributions are replayed, with every literal author and committer header, date, time zone, encoding, message and contributor trailer preserved.

The third contribution's relation-helper signature overlapped upstream's builtin dictionary annotations. The composition keeps the complete original room_id signature and callers, together with the upstream dict annotations. The complete adapter AST otherwise equals the preceding published own layer after those annotation conversions. The relation module and the complete live-test fixture and thread-test files are byte-identical to the original own layer. Every original unit-test assertion and parameter table remains; three import statements use their actual upstream forms. All paths outside this layer's five changed files equal the exact published prerequisite.

Current signed head: d6f4b2428e1523c116cc67451b8a36b95eda32c4. Source tree: 9f58ee20f6049723f0d39172714151755ee2ab29. Review the seven-contribution layer against the exact published prerequisite.

Fresh canonical validation passes 221 cases across three complete files: Matrix adapter (210), Docker endpoint configuration (7) and gateway readiness fixtures (4). There are no failures, skips or retries. The gate uses the real saved SDK, isolated test homes, strict acceptance and runtime, thread and unraisable-exception warnings as errors. Earlier native Synapse exchanges and broader combined gates retain their recorded source revisions and scopes. This gate does not establish a fresh native Matrix exchange or hosted-CI acceptance.

Type checks remain non-zero with 52 findings in each of the original, prerequisite and candidate five-path scopes. Complete diagnostic descriptions, reported source spans and columns, whole statement ASTs and multiplicity Counters introduce zero findings against those actual source operands. Ruff and changed-source whitespace checks pass. The same four inherited files require formatting, with no new formatter path. The exact local join with the required upstream operand is clean. The PR remains a draft.

Merge fix/matrix-effective-event-state at 666a75b (NousResearch#126282). It now includes NousResearch#125735, NousResearch#125697, NousResearch#125660, NousResearch#125648, NousResearch#62088, NousResearch#51804, NousResearch#128621 and main at 02e4118. Conflicts: - plugins/platforms/matrix/adapter.py: main removed the unused EA_HEADER_TEXT import. Take the removal and keep the room inspection import. - tools/matrix_read_tool.py: the parent schedules every read with safe_schedule_threadsafe and a named deadline, and it drops the same-loop branch. Inspection kinds go through the same scheduling path, and history kinds call the parent's read_context. The tool description keeps this layer's list of kinds, followed by the parent's sentences about event order, skipped events and errors. - tests/integration/matrix_live/conftest.py: the gateway runs as the host user and the model server binds to the host route. The inspection mode keeps its empty script and its default reply, now taken from GatewaySettings.reply. - tests/gateway/test_matrix_read_context.py: take the parent's _client and _adapter helpers in each hunk. - tests/gateway/test_matrix_reaction_context.py: import _make_adapter and the parent's _rendered together. - tests/tools/test_matrix_read_tool.py: keep this layer's inspection test and the parent's parametrised argument and session tests. Later commits adapt the tests and the inspection module to the parent's interfaces.

The effective-state parent now contains NousResearch#125648, which quotes the triggering event once per threaded response. A send anchored on an event that Hermes has already answered in the thread is a continuation: it sets is_falling_back and points at the latest event in the thread. A correction that waits behind a paused turn runs as the last message of the queued chain. Its answer is anchored on the message that started the paused turn, as for every queued message, and the first response already answered that message. The native test still expected both responses to be genuine replies to that message, so the plain and encrypted enabled cases failed with is_falling_back true and m.in_reply_to pointing at the first response. The client now expects the first response to quote the blocked message and each later response to fall back to the response before it, in the expected order. The Matrix guide describes the correction's reply in that case as a thread message after the running turn's answer.

Merge fix/matrix-effective-event-state at 666a75b (NousResearch#126282). It now includes NousResearch#125735, NousResearch#125697, NousResearch#125660, NousResearch#125648, NousResearch#62088, NousResearch#51804, NousResearch#128621 and main at 02e4118. This branch still carries room inspection at 8b87573 and receipts lifecycle at f2c50f7. Conflicts: - gateway/platforms/event.py: keep the receipts processing state and voice preparation fields beside the parent's merged message IDs and channel state, and both TYPE_CHECKING imports. absorb_media and authored_media sit beside absorb_turn_input and receipt_message_id. - gateway/platforms/base.py: the three merge paths take the parent's absorb_media and absorb_message_ids and drop append_channel_context, which the parent removed. absorb_turn_input replaces absorb_reply_expected, because it calls it and also moves the receipt target. - gateway/run_busy.py: main translated the busy and steer replies. Keep the translations with the receipts layer's approval outcome hook, its finished-turn steer fallback and input_text. The approval routing reads main's per-language word list. /queue keeps the receipts layer's dataclasses.replace copy, so it shares the processing state and carries the parent's quoted-media and inbound-context dependencies; the merged message IDs are copied as a new list, like the media lists. - gateway/run_turn.py: keep processing_event beside the parent's input_snapshot and main's title_user_message, and record the agent outcome before the parent copies the prepared input back. - gateway/run_inbound.py: re-home inbound media first, then read the voice preparation text. - hermes_cli/config_defaults.py: keep room_backfill_limit and the receipt and reaction defaults. - plugins/platforms/matrix/adapter.py: main removed the unused EA_HEADER_TEXT import. Keep the inspection, feedback, session and room-context imports; the immediate-receipt gate with the parent's seven-field context; receipt_event_id and reply_parent as keyword arguments of _handle_media_message; and inspect_matrix_room. The parent removed fetch_thread_context. - tools/matrix_read_tool.py and tests/tools/test_matrix_read_tool.py: resolved as on the room inspection branch. Inspection kinds use the parent's scheduling path, and both tests are kept. - tests/integration/matrix_live/conftest.py: take the foundation's host user, host route and HOME, and the parent's modes and plugins, with the receipts layer's feedback settings, busy mode, approvals and delivery probe. The model_responder fixture now defaults to None, and the gateway then scripts GatewaySettings.reply as the parent does. A test that overrides model_responder keeps its callable. The queued context mode keeps its queue busy mode; every other mode writes the receipts layer's busy settings. - tests/integration/matrix_live/test_read_tools.py: keep the inspection test and the parent's encrypted edit test.

Merge feat/matrix-unread-base at 2a4c4e6. It now includes the effective-state parent at 666a75b (NousResearch#126282), with NousResearch#125735, NousResearch#125697, NousResearch#125660, NousResearch#125648, NousResearch#62088, NousResearch#51804, NousResearch#128621 and main at 02e4118. Conflicts: - hermes_cli/tools_config.py and hermes_cli/toolset_scope.py: the parent adds matrix_read to CONFIGURABLE_TOOLSETS and restricts it to Matrix. Keep matrix_unread beside it in both. - plugins/platforms/matrix/read_context.py: take the parent's reader. It decrypts through client_events, so this layer's _decrypted_event and _raw_event are gone; a later commit moves unread.py to the client_events helpers. - plugins/platforms/matrix/adapter.py: main removed the unused EA_HEADER_TEXT import, and the parent removed fetch_thread_context. Keep the unread import and the read_matrix_unread and mark_matrix_read methods. - tests/integration/matrix_live/conftest.py: take the parent's host route, host user and model script. Its model_responder defaults to None and the gateway then scripts GatewaySettings.reply, which is "Matrix live reply" by default, so the unread test's first reply is unchanged and later replies come from its pushed script. Keep this layer's gateway_auxiliary_config fixture.

The effective-state parent now contains NousResearch#125648, which quotes the triggering event once per threaded response. A send anchored on an event that Hermes has already answered in the thread is a continuation: it sets is_falling_back and points at the latest event in the thread. A correction that waits behind a paused turn runs as the last message of the queued chain. Its answer is anchored on the message that started the paused turn, as for every queued message, and the first response already answered that message. The native test still expected both responses to be genuine replies to that message, so the plain and encrypted enabled cases failed with is_falling_back true and m.in_reply_to pointing at the first response. The client now expects the first response to quote the blocked message and each later response to fall back to the response before it, in the expected order. The Matrix guide describes the correction's reply in that case as a thread message after the running turn's answer.

The DM tests marked m.direct account data but did not supply joined members. The two-person classifier therefore treated those fixtures as rooms, so their threading assertions tested the wrong policy. Give the fixture a full joined member list with the bot and one other user. Keep the three-member variant for room behaviour. The mention and threading tests now exercise the intended DM policy.

A room is a private bot chat only when the bot's user ID is one of its two joined members. With an access token, the adapter replaces the configured MATRIX_USER_ID with the ID that whoami returns. After a password login it kept the configured string. Synapse accepts a login identifier whose localpart differs in case from the registered user, so a configured @Hermes:example.org logs in as @Hermes:example.org. The membership check then never matched, and every one-to-one chat followed room rules: with MATRIX_ALLOWED_ROOMS set its messages were dropped, and otherwise they needed a mention. mautrix's login stores the homeserver's user ID in client.mxid. Copy it into the adapter after a successful login, as the whoami path does, so the membership check uses the ID that appears in the room's member list.

The Matrix handoff session-key test marked its DM case only through the m.direct cache. Room classification no longer reads m.direct, so the DM case was keyed as a group and the test failed. Give the fake client a joined member list: the bot and one other user for the DM case, and a third user for the room case.

The environment-variable reference still said that any room with two or fewer joined members is a DM, and told operators to add a third member to get room behaviour. A bot-only room, a room without the bot and a room whose membership cannot be read now follow room rules. Describe the current rule in the MATRIX_ALLOWED_ROOMS, MATRIX_REQUIRE_MENTION, MATRIX_FREE_RESPONSE_ROOMS and MATRIX_AUTO_THREAD rows. The comment in _schedule_pending_invite_joins said that a direct invite missing from m.direct is misclassified as a group. m.direct no longer affects classification, so drop that clause.

The E2EE crypto store keys every row (the Olm account, Olm sessions and inbound Megolm sessions) by an account ID taken from the adapter's user ID. After a password login the adapter now replaces the configured MATRIX_USER_ID with the homeserver's spelling of it, so the DM check matches the room's member list. That replacement also changed the store key. A deployment with a password login, MATRIX_USER_ID differing in case from the registered user, and a stable MATRIX_DEVICE_ID would open an empty store after upgrading. It would create a new Olm account for its existing device and could no longer decrypt history. The same happened on an in-process reconnect, because the second login read the replaced ID. Keep the configured ID in its own attribute and use it for the login identifier and, after a password login, for the store key, as before. The access-token path keys the store by the whoami answer, as it always has. The adapter's user ID still becomes the homeserver's spelling on both paths, and the DM, mention, invite and self-sender checks use it.

Matrix rooms often have no m.room.name state event. Identity resolution used the opaque room ID as the name shown to the agent even when other occupants had readable display names. Read member profiles from the state store, with the joined members API as a fallback when the store is empty. Use the other occupants to title an unnamed room. Explicit-name detection and DM classification remain unchanged.

Matrix room identity was cached without reacting to live state changes. Topic edits, renames and membership changes could leave stale room context. The agent also missed important privacy or replacement changes. Register sync-awaited state handlers that invalidate the identity cache. Queue one note per change kind for the next ordinary message. Keep notes through mention drops, commands and text batching. Quote and bound room-state values before they enter channel context.

Room notes were consumed while the adapter built an event. The gateway could reject that event for sender policy or intercept it for a paused session, leaving no note for the next model turn. A queued follow-up also bypasses normal admission and missed the note entirely. Consume notes in inbound model-message preparation, which both cold and queued turns use after their dispatch gates. Rejected and intercepted messages therefore leave pending notes for the next turn. Busy-session debounce and pending-message merges also discarded channel context from later events. Append it during both merges so accepted context reaches the model. Request joined member profiles explicitly so invited users do not appear in unnamed room titles. Runner admission, paused and queued turns, busy merges, room bounds, coalescing and invited membership have focused regressions.

The effective-state parent now contains NousResearch#125648, which quotes the triggering event once per threaded response. A send anchored on an event that Hermes has already answered in the thread is a continuation: it sets is_falling_back and points at the latest event in the thread. A correction that waits behind a paused turn runs as the last message of the queued chain. Its answer is anchored on the message that started the paused turn, as for every queued message, and the first response already answered that message. The native test still expected both responses to be genuine replies to that message, so the plain and encrypted enabled cases failed with is_falling_back true and m.in_reply_to pointing at the first response. The client now expects the first response to quote the blocked message and each later response to fall back to the response before it, in the expected order. The Matrix guide describes the correction's reply in that case as a thread message after the running turn's answer.

The effective-state parent now contains NousResearch#125648, which quotes the triggering event once per threaded response. A send anchored on an event that Hermes has already answered in the thread is a continuation: it sets is_falling_back and points at the latest event in the thread. A correction that waits behind a paused turn runs as the last message of the queued chain. Its answer is anchored on the message that started the paused turn, as for every queued message, and the first response already answered that message. The native test still expected both responses to be genuine replies to that message, so the plain and encrypted enabled cases failed with is_falling_back true and m.in_reply_to pointing at the first response. The client now expects the first response to quote the blocked message and each later response to fall back to the response before it, in the expected order. The Matrix guide describes the correction's reply in that case as a thread message after the running turn's answer. (cherry picked from commit c3dec29d8ca671ceeb269ee4b0f589a3dad390d7)