refactor(matrix): migrate from matrix-nio to mautrix-python by teknium1 · Pull Request #7518 · NousResearch/hermes-agent

teknium1 · 2026-04-11T03:02:49Z

Summary

Salvage of PR #7488 by @alt-glitch — full migration of the Matrix gateway adapter from the unmaintained matrix-nio SDK to mautrix-python.

What this PR does: Replaces matrix-nio with mautrix-python as the Matrix SDK dependency. Fixes Nix flake build failures caused by matrix-nio's transitive dep atomicwrites (sdist-only, archived). Preserves all adapter features: E2EE, reactions, threading, mention gating, text batching, media caching, MSC3245 voice messages.

Why move away from matrix-nio?

Unmaintained — last meaningful release was over a year ago
Heavy, problematic dependency tree (peewee, atomicwrites)
macOS build failures with python-olm/libolm
No wheels published (sdist-only)
mautrix-python is actively maintained by the author of major Matrix bridges

Changes from the original PR

All 8 contributor commits cherry-picked with authorship preserved. One follow-up commit:

Module-level import guard: Wrapped from mautrix.types import ... in try/except with proper stub classes so the module is importable without mautrix. Without this, the gateway crashes for ALL platforms when mautrix isn't installed.
Test isolation fixes: Removed _ensure_mautrix_mock() which permanently polluted sys.modules. Fixed thread persistence tests to use direct class references. Moved module-importability test to subprocess to prevent module reload pollution.

Test plan

152 matrix-related tests pass (test_matrix, test_matrix_mention, test_setup_matrix_e2ee, test_project_metadata)
Module importability verified without mautrix SDK
Closes refactor(matrix): migrate from matrix-nio to mautrix-python #7488 — contributor commits preserved in git log

Credit: @alt-glitch for the full migration implementation.

matrix-nio pulls in peewee -> atomicwrites (sdist-only, archived, missing build-system metadata) which breaks nix flake builds. mautrix-python publishes wheels, has a leaner dep tree, and its [encryption] extra uses the same python-olm without the problematic transitive chain.

Translate all nio SDK calls to mautrix equivalents while preserving the adapter structure, business logic, and all features (E2EE, reactions, threading, mention gating, text batching, media caching, voice MSC3245). Key changes: - nio.AsyncClient -> mautrix.client.Client + HTTPAPI + MemoryStateStore - Manual E2EE key management -> OlmMachine with auto key lifecycle - isinstance(resp, nio.XxxResponse) -> mautrix returns values directly - add_event_callback per type -> single ROOM_MESSAGE handler with msgtype dispatch - Room state (member_count, display_name) via async state store lookups - Upload/download return ContentURI/bytes directly (no wrapper objects)

Rewrite mock infrastructure across three test files: - test_matrix.py: replace fake nio module with fake mautrix module tree, update all client method mocks to new API names and return types - test_matrix_voice.py: update event construction, download/upload mocks, handler invocation (single event arg, no room object) - test_matrix_mention.py: update mock module, event construction, DM detection via _dm_rooms cache instead of room.member_count 157 tests passing.

Address two bugs found by code review: 1. MemoryCryptoStore loses all E2EE keys on restart — now pickle the store to disk on disconnect and restore on connect, preserving Megolm sessions across restarts. 2. Encrypted events buffered for retry were silently dropped after decryption because _on_encrypted_event registered the event ID in the dedup set, then _on_room_message rejected it as a duplicate. Now clear the dedup entry before routing decrypted events.

- Extract _resolve_message_context() to deduplicate ~40 lines of mention/thread/DM gating logic between text and media handlers - Move mautrix.types imports to module level (16 scattered local imports consolidated) - Parse mention/thread env vars once in __init__ instead of per-message - Cache _is_bot_mentioned() result instead of calling 3x per event - Consolidate send_emote/send_notice into shared _send_simple_message() - Use _is_dm_room() in get_chat_info() instead of inline duplication - Add _CRYPTO_PICKLE_PATH constant (was duplicated in 2 locations) - Fix fragile event_ts extraction (double getattr, None safety) - Clean up leaked aiohttp session on auth failure paths - Remove redundant trailing _track_thread() calls

…kle store - Add api.session.close() on E2EE dep check and E2EE setup failure paths (two missing cleanup points from the mautrix migration) - Replace raw pickle.load/dump with HMAC-SHA256 signed payloads to prevent arbitrary code execution from a tampered store file

The old nio code only handled RoomMessageText (m.text). The mautrix rewrite dispatched both m.text and m.notice, which would cause infinite loops between bots since m.notice is the conventional msgtype for bot responses in the Matrix ecosystem.

…tion Follow-up fixes for the matrix-nio → mautrix migration: 1. Module-level mautrix.types import now wrapped in try/except with proper stub classes. Without this, importing gateway.platforms.matrix crashes the entire gateway when mautrix isn't installed — even for users who don't use Matrix. The stubs mirror mautrix's real attribute names so tests that exercise adapter methods (send, reactions, etc.) work without the real SDK. 2. Removed _ensure_mautrix_mock() from test_matrix_mention.py — it permanently installed MagicMock modules in sys.modules via setdefault(), polluting later tests in the suite. No longer needed since the module imports cleanly without mautrix. 3. Fixed thread persistence tests to use direct class reference in monkeypatch.setattr() instead of string-based paths, which broke when the module was reimported by other tests. 4. Moved the module-importability test to a subprocess to prevent it from polluting sys.modules (reimporting creates a second module object with different __dict__, breaking patch.object in subsequent tests).

github-actions · 2026-04-11T03:03:04Z

⚠️ Supply Chain Risk Detected

This PR contains patterns commonly associated with supply chain attacks. This does not mean the PR is malicious — but these patterns require careful human review before merging.

⚠️ WARNING: Install hook files modified

These files can execute code during package installation or interpreter startup.

Files:

hermes_cli/setup.py

⚠️ WARNING: marshal/pickle/compile usage

These can deserialize or construct executable code objects.

Matches:

371:+                                saved = pickle.loads(payload)  # noqa: S301

Automated scan triggered by supply-chain-audit. If this is a false positive, a maintainer can approve after manual review.

benhaotang · 2026-04-11T09:20:45Z

After I pulled the main branch, the hermes gateway cannot start anymore

ERROR gateway.platforms.matrix: Matrix: failed to create E2EE client: MemoryCryptoStore.__init__() missing 2 required positional arguments: 'account_id' and 'pickle_key'. Install with: pip install 'mautrix[encryption]'  (requires libolm C library)
WARNING gateway.run: ✗ matrix failed to connect
ERROR gateway.run: Gateway failed to connect any configured messaging platform: matrix: failed to connect

Already had mautrix[encryption] installed into the venv of hermes (can be seen by uv pip list --python ~/.hermes/hermes-agent/venv/bin/python)

Tried to reconfigure matrix gateway, restart the gateway, still this error. Is there anything we need to change on our end in config?

I currently have to git check out to the commit before this PR (a8fd725) to make matrix connect again.

Three bugs that break encrypted communication after the matrix-nio to mautrix-python migration (#7518): 1. Silent device key upload failure: after nuking crypto.db, share_keys() returns HTTP 200 but the server silently ignores new device keys when identity keys are immutable for the existing device. The bot sets shared=True and proceeds, but other clients encrypt to the old key. Fix: re-verify server keys after share_keys() and fail-closed on mismatch. 2. Dedup race between two ROOM_ENCRYPTED handlers: DecryptionDispatcher (auto-registered by mautrix) and _on_encrypted_event (manual) both fire for every encrypted event. _on_encrypted_event has zero awaits and marks the event_id in the dedup set before DecryptionDispatcher finishes decrypting, causing _on_room_message to drop the decrypted event. Fix: replace both with a single _HermesDecryptionDispatcher that handles decrypt success and failure in one path. 3. _joined_rooms reference invalidation: _CryptoStateStore captures self._joined_rooms by reference at init, but connect() reassigns self._joined_rooms = set(...) after initial sync, orphaning the reference. find_shared_rooms() returns [] forever. Fix: mutate in-place with clear()+update(). Also removes debug instrumentation added during investigation. Closes #8174

djmaze · 2026-04-14T22:52:51Z

For me, the gateway starts and the matrix connection seems to be working, but the bot does not react to anything. It does not even accept invitations, even in non-encrypted rooms. I even tried with a new bot user and removing the crypto db.

Since the matrix-nio → mautrix-python migration (NousResearch#7518), InternalEventType.INVITE was never dispatched because the MembershipEventDispatcher — which converts raw ROOM_MEMBER state events into INVITE/JOIN/LEAVE internal events — was not registered on the client. The bot connected and synced but silently ignored all room invites, so it never joined rooms or processed messages. Register MembershipEventDispatcher in connect() and add the corresponding stub to the test mock modules. Fixes NousResearch#10725 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fxfitz · 2026-04-16T04:03:09Z

I opened up this issue: #10725
And then this PR: #10732

Fixes NousResearch#10725 Root cause: Since migration from matrix-nio to mautrix-python in NousResearch#7518, the MembershipEventDispatcher was not registered on the mautrix Client instance. In mautrix-python, InternalEventType.INVITE is not dispatched directly by handle_sync(). The MembershipEventDispatcher catches EventType.ROOM_MEMBER state events and re-dispatches them as InternalEventType.INVITE, JOIN, LEAVE, etc. Without this dispatcher, the INVITE handler never fired, so the bot never auto-joined rooms and never received messages. Fix: Register MembershipEventDispatcher on the client before adding event handlers. Also updated test mock to include the dispatcher module and add_dispatcher method.

The mautrix migration (#7518) broke auto-join because InternalEventType.INVITE events are only dispatched when MembershipEventDispatcher is registered on the client. Without it, _on_invite is dead code and the bot silently ignores all room invites. Closes #10094 Closes #10725 Refs: PR #10135 (digging-airfare-4u), PR #10732 (fxfitz)

* - make buffered streaming - fix path naming to expand `~` for agent. - fix stripping of matrix ID to not remove other mentions / localports. * fix(matrix): register MembershipEventDispatcher for invite auto-join The mautrix migration (#7518) broke auto-join because InternalEventType.INVITE events are only dispatched when MembershipEventDispatcher is registered on the client. Without it, _on_invite is dead code and the bot silently ignores all room invites. Closes #10094 Closes #10725 Refs: PR #10135 (digging-airfare-4u), PR #10732 (fxfitz) * fix(matrix): preserve _joined_rooms reference for CryptoStateStore connect() reassigned self._joined_rooms = set(...) after initial sync, orphaning the reference captured by _CryptoStateStore at init time. find_shared_rooms() returned [] forever, breaking Megolm session rotation on membership changes. Mutate in place with clear() + update() so the CryptoStateStore reference stays valid. Refs #8174, PR #8215 * fix(matrix): remove dual ROOM_ENCRYPTED handler to fix dedup race mautrix auto-registers DecryptionDispatcher when client.crypto is set. The adapter also registered _on_encrypted_event for the same event type. _on_encrypted_event had zero awaits and won the race to mark event IDs in the dedup set, causing _on_room_message to drop successfully decrypted events from DecryptionDispatcher. The retry loop masked this by re-decrypting every message ~4 seconds later. Remove _on_encrypted_event entirely. DecryptionDispatcher handles decryption; genuinely undecryptable events are logged by mautrix and retried on next key exchange. Refs #8174, PR #8215 * fix(matrix): re-verify device keys after share_keys() upload Matrix homeservers treat ed25519 identity keys as immutable per device. share_keys() can return 200 but silently ignore new keys if the device already exists with different identity keys. The bot would proceed with shared=True while peers encrypt to the old (unreachable) keys. Now re-queries the server after share_keys() and fails closed if keys don't match, with an actionable error message. Refs #8174, PR #8215 * fix(matrix): encrypt outbound attachments in E2EE rooms _upload_and_send() uploaded raw bytes and used the 'url' key for all rooms. In E2EE rooms, media must be encrypted client-side with encrypt_attachment(), the ciphertext uploaded, and the 'file' key (with key/iv/hashes) used instead of 'url'. Now detects encrypted rooms via state_store.is_encrypted() and branches to the encrypted upload path. Refs: PR #9822 (charles-brooks) * fix(matrix): add stop_typing to clear typing indicator after response The adapter set a 30-second typing timeout but never cleared it. The base class stop_typing() is a no-op, so the typing indicator lingered for up to 30 seconds after each response. Closes #6016 Refs: PR #6020 (r266-tech) * fix(matrix): cache all media types locally, not just photos/voice should_cache_locally only covered PHOTO, VOICE, and encrypted media. Unencrypted audio/video/documents in plaintext rooms were passed as MXC URLs that require authentication the agent doesn't have, resulting in 401 errors. Refs #3487, #3806 * fix(matrix): detect stale OTK conflict on startup and fail closed When crypto state is wiped but the same device ID is reused, the homeserver may still hold one-time keys signed with the previous identity key. Identity key re-upload succeeds but OTK uploads fail with "already exists" and a signature mismatch. Peers cannot establish new Olm sessions, so all new messages are undecryptable. Now proactively flushes OTKs via share_keys() during connect() and catches the "already exists" error with an actionable log message telling the operator to purge the device from the homeserver or generate a fresh device ID. Also documents the crypto store recovery procedure in the Matrix setup guide. Refs #8174 * docs(matrix): improve crypto recovery docs per review - Put easy path (fresh access token) first, manual purge second - URL-encode user ID in Synapse admin API example - Note that device deletion may invalidate the access token - Add "stop Synapse first" caveat for direct SQLite approach - Mention the fail-closed startup detection behavior - Add back-reference from upgrade section to OTK warning * refactor(matrix): cleanup from code review - Extract _extract_server_ed25519() and _reverify_keys_after_upload() to deduplicate the re-verification block (was copy-pasted in two places, three copies of ed25519 key extraction total) - Remove dead code: _pending_megolm, _retry_pending_decryptions, _MAX_PENDING_EVENTS, _PENDING_EVENT_TTL — all orphaned after removing _on_encrypted_event - Remove tautological TestMediaCacheGate (tested its own predicate, not production code) - Remove dead TestMatrixMegolmEventHandling and TestMatrixRetryPendingDecryptions (tested removed methods) - Merge duplicate TestMatrixStopTyping into TestMatrixTypingIndicator - Trim comment to just the "why"

Three bugs that break encrypted communication after the matrix-nio to mautrix-python migration (NousResearch#7518): 1. Silent device key upload failure: after nuking crypto.db, share_keys() returns HTTP 200 but the server silently ignores new device keys when identity keys are immutable for the existing device. The bot sets shared=True and proceeds, but other clients encrypt to the old key. Fix: re-verify server keys after share_keys() and fail-closed on mismatch. 2. Dedup race between two ROOM_ENCRYPTED handlers: DecryptionDispatcher (auto-registered by mautrix) and _on_encrypted_event (manual) both fire for every encrypted event. _on_encrypted_event has zero awaits and marks the event_id in the dedup set before DecryptionDispatcher finishes decrypting, causing _on_room_message to drop the decrypted event. Fix: replace both with a single _HermesDecryptionDispatcher that handles decrypt success and failure in one path. 3. _joined_rooms reference invalidation: _CryptoStateStore captures self._joined_rooms by reference at init, but connect() reassigns self._joined_rooms = set(...) after initial sync, orphaning the reference. find_shared_rooms() returns [] forever. Fix: mutate in-place with clear()+update(). Also removes debug instrumentation added during investigation. Closes NousResearch#8174

* - make buffered streaming - fix path naming to expand `~` for agent. - fix stripping of matrix ID to not remove other mentions / localports. * fix(matrix): register MembershipEventDispatcher for invite auto-join The mautrix migration (NousResearch#7518) broke auto-join because InternalEventType.INVITE events are only dispatched when MembershipEventDispatcher is registered on the client. Without it, _on_invite is dead code and the bot silently ignores all room invites. Closes NousResearch#10094 Closes NousResearch#10725 Refs: PR NousResearch#10135 (digging-airfare-4u), PR NousResearch#10732 (fxfitz) * fix(matrix): preserve _joined_rooms reference for CryptoStateStore connect() reassigned self._joined_rooms = set(...) after initial sync, orphaning the reference captured by _CryptoStateStore at init time. find_shared_rooms() returned [] forever, breaking Megolm session rotation on membership changes. Mutate in place with clear() + update() so the CryptoStateStore reference stays valid. Refs NousResearch#8174, PR NousResearch#8215 * fix(matrix): remove dual ROOM_ENCRYPTED handler to fix dedup race mautrix auto-registers DecryptionDispatcher when client.crypto is set. The adapter also registered _on_encrypted_event for the same event type. _on_encrypted_event had zero awaits and won the race to mark event IDs in the dedup set, causing _on_room_message to drop successfully decrypted events from DecryptionDispatcher. The retry loop masked this by re-decrypting every message ~4 seconds later. Remove _on_encrypted_event entirely. DecryptionDispatcher handles decryption; genuinely undecryptable events are logged by mautrix and retried on next key exchange. Refs NousResearch#8174, PR NousResearch#8215 * fix(matrix): re-verify device keys after share_keys() upload Matrix homeservers treat ed25519 identity keys as immutable per device. share_keys() can return 200 but silently ignore new keys if the device already exists with different identity keys. The bot would proceed with shared=True while peers encrypt to the old (unreachable) keys. Now re-queries the server after share_keys() and fails closed if keys don't match, with an actionable error message. Refs NousResearch#8174, PR NousResearch#8215 * fix(matrix): encrypt outbound attachments in E2EE rooms _upload_and_send() uploaded raw bytes and used the 'url' key for all rooms. In E2EE rooms, media must be encrypted client-side with encrypt_attachment(), the ciphertext uploaded, and the 'file' key (with key/iv/hashes) used instead of 'url'. Now detects encrypted rooms via state_store.is_encrypted() and branches to the encrypted upload path. Refs: PR NousResearch#9822 (charles-brooks) * fix(matrix): add stop_typing to clear typing indicator after response The adapter set a 30-second typing timeout but never cleared it. The base class stop_typing() is a no-op, so the typing indicator lingered for up to 30 seconds after each response. Closes NousResearch#6016 Refs: PR NousResearch#6020 (r266-tech) * fix(matrix): cache all media types locally, not just photos/voice should_cache_locally only covered PHOTO, VOICE, and encrypted media. Unencrypted audio/video/documents in plaintext rooms were passed as MXC URLs that require authentication the agent doesn't have, resulting in 401 errors. Refs NousResearch#3487, NousResearch#3806 * fix(matrix): detect stale OTK conflict on startup and fail closed When crypto state is wiped but the same device ID is reused, the homeserver may still hold one-time keys signed with the previous identity key. Identity key re-upload succeeds but OTK uploads fail with "already exists" and a signature mismatch. Peers cannot establish new Olm sessions, so all new messages are undecryptable. Now proactively flushes OTKs via share_keys() during connect() and catches the "already exists" error with an actionable log message telling the operator to purge the device from the homeserver or generate a fresh device ID. Also documents the crypto store recovery procedure in the Matrix setup guide. Refs NousResearch#8174 * docs(matrix): improve crypto recovery docs per review - Put easy path (fresh access token) first, manual purge second - URL-encode user ID in Synapse admin API example - Note that device deletion may invalidate the access token - Add "stop Synapse first" caveat for direct SQLite approach - Mention the fail-closed startup detection behavior - Add back-reference from upgrade section to OTK warning * refactor(matrix): cleanup from code review - Extract _extract_server_ed25519() and _reverify_keys_after_upload() to deduplicate the re-verification block (was copy-pasted in two places, three copies of ed25519 key extraction total) - Remove dead code: _pending_megolm, _retry_pending_decryptions, _MAX_PENDING_EVENTS, _PENDING_EVENT_TTL — all orphaned after removing _on_encrypted_event - Remove tautological TestMediaCacheGate (tested its own predicate, not production code) - Remove dead TestMatrixMegolmEventHandling and TestMatrixRetryPendingDecryptions (tested removed methods) - Merge duplicate TestMatrixStopTyping into TestMatrixTypingIndicator - Trim comment to just the "why"

alt-glitch and others added 9 commits April 10, 2026 19:48

docs(matrix): update all references from matrix-nio to mautrix

732785a

teknium1 merged commit be9198f into main Apr 11, 2026
5 of 7 checks passed

nousr mentioned this pull request Apr 11, 2026

feat(nix): container-aware CLI — auto-route into managed container #7543

Merged

5 tasks

alt-glitch mentioned this pull request Apr 12, 2026

fix(gateway): fix Matrix E2EE silent failures after mautrix migration #8215

Closed

4 tasks

yzx9 mentioned this pull request Apr 14, 2026

fix(setup): graceful fallback when matrix-nio[e2e] install fails #5936

Closed

github-actions Bot mentioned this pull request Apr 15, 2026

chore: bump NousResearch/hermes-agent version from v2026.4.8 to v2026.4.13 Docker-Hub-sirmark/docker-hermes-agent#1

Merged

This was referenced Apr 16, 2026

fix(matrix): register MembershipEventDispatcher for invite auto-join #10725

Closed

fix(matrix): register MembershipEventDispatcher for invite auto-join #10732

Closed

nightq mentioned this pull request Apr 16, 2026

fix(matrix): register MembershipEventDispatcher for invite auto-join #10803

Closed

2 tasks

alt-glitch mentioned this pull request Apr 16, 2026

fix(matrix): E2EE and migration bugfixes #10860

Merged

6 tasks

alt-glitch mentioned this pull request May 16, 2026

refactor(matrix): migrate from matrix-nio to mautrix-python #27095

Closed

4 tasks

phgw-agent mentioned this pull request Jul 24, 2026

Matrix bot silently drops ALL inbound messages in encrypted rooms it already joined (fresh crypto store): _CryptoStateStore.get_encryption_info never consults the homeserver #71067

Open

Navigation Menu

Uh oh!

refactor(matrix): migrate from matrix-nio to mautrix-python - #7518

refactor(matrix): migrate from matrix-nio to mautrix-python#7518
teknium1 merged 9 commits into
mainfrom
hermes/hermes-fefa061a

teknium1 commented Apr 11, 2026

Uh oh!

github-actions Bot commented Apr 11, 2026

Uh oh!

Uh oh!

benhaotang commented Apr 11, 2026

Uh oh!

djmaze commented Apr 14, 2026

Uh oh!

fxfitz commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

teknium1 commented Apr 11, 2026

Summary

Changes from the original PR

Test plan

Uh oh!

github-actions Bot commented Apr 11, 2026

⚠️ Supply Chain Risk Detected

⚠️ WARNING: Install hook files modified

⚠️ WARNING: marshal/pickle/compile usage

Uh oh!

Uh oh!

benhaotang commented Apr 11, 2026

Uh oh!

djmaze commented Apr 14, 2026

Uh oh!

fxfitz commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants