fix(matrix): replace pickle crypto store with SQLite, fix E2EE decryption - #7981
Merged
Conversation
…tion Fixes #7952 — Matrix E2EE completely broken after mautrix migration. - Replace MemoryCryptoStore + pickle/HMAC persistence with mautrix's PgCryptoStore backed by SQLite via aiosqlite. Crypto state now persists reliably across restarts without fragile serialization. - Add handle_sync() call on initial sync response so to-device events (queued Megolm key shares) are dispatched to OlmMachine instead of being silently dropped. - Add _verify_device_keys_on_server() after loading crypto state. Detects missing keys (re-uploads), stale keys from migration (attempts re-upload), and corrupted state (refuses E2EE). - Add _CryptoStateStore adapter wrapping MemoryStateStore to satisfy mautrix crypto's StateStore interface (is_encrypted, get_encryption_info, find_shared_rooms). - Remove redundant share_keys() call from sync loop — OlmMachine already handles this via DEVICE_OTK_COUNT event handler. - Fix datetime vs float TypeError in session.py suspend_recently_active() that crashed gateway startup. - Add aiosqlite and asyncpg to [matrix] extra in pyproject.toml. - Update test mocks for PgCryptoStore/Database and add query_keys mock for key verification. 174 tests pass. - Add E2EE upgrade/migration docs to Matrix user guide.
1 task
Contributor
|
confirming it works for me 🎉 (my local qwen3.5 had been pointing at the same pickling thing) |
Contributor
|
Still borked for me. Is there a way to just nuke the previously configured matrix stuff into the sun and start fresh? |
1 task
Tommyeds
pushed a commit
to Tommyeds/hermes-agent
that referenced
this pull request
Apr 12, 2026
…tion (NousResearch#7981) Fixes NousResearch#7952 — Matrix E2EE completely broken after mautrix migration. - Replace MemoryCryptoStore + pickle/HMAC persistence with mautrix's PgCryptoStore backed by SQLite via aiosqlite. Crypto state now persists reliably across restarts without fragile serialization. - Add handle_sync() call on initial sync response so to-device events (queued Megolm key shares) are dispatched to OlmMachine instead of being silently dropped. - Add _verify_device_keys_on_server() after loading crypto state. Detects missing keys (re-uploads), stale keys from migration (attempts re-upload), and corrupted state (refuses E2EE). - Add _CryptoStateStore adapter wrapping MemoryStateStore to satisfy mautrix crypto's StateStore interface (is_encrypted, get_encryption_info, find_shared_rooms). - Remove redundant share_keys() call from sync loop — OlmMachine already handles this via DEVICE_OTK_COUNT event handler. - Fix datetime vs float TypeError in session.py suspend_recently_active() that crashed gateway startup. - Add aiosqlite and asyncpg to [matrix] extra in pyproject.toml. - Update test mocks for PgCryptoStore/Database and add query_keys mock for key verification. 174 tests pass. - Add E2EE upgrade/migration docs to Matrix user guide.
1 task
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
…tion (NousResearch#7981) Fixes NousResearch#7952 — Matrix E2EE completely broken after mautrix migration. - Replace MemoryCryptoStore + pickle/HMAC persistence with mautrix's PgCryptoStore backed by SQLite via aiosqlite. Crypto state now persists reliably across restarts without fragile serialization. - Add handle_sync() call on initial sync response so to-device events (queued Megolm key shares) are dispatched to OlmMachine instead of being silently dropped. - Add _verify_device_keys_on_server() after loading crypto state. Detects missing keys (re-uploads), stale keys from migration (attempts re-upload), and corrupted state (refuses E2EE). - Add _CryptoStateStore adapter wrapping MemoryStateStore to satisfy mautrix crypto's StateStore interface (is_encrypted, get_encryption_info, find_shared_rooms). - Remove redundant share_keys() call from sync loop — OlmMachine already handles this via DEVICE_OTK_COUNT event handler. - Fix datetime vs float TypeError in session.py suspend_recently_active() that crashed gateway startup. - Add aiosqlite and asyncpg to [matrix] extra in pyproject.toml. - Update test mocks for PgCryptoStore/Database and add query_keys mock for key verification. 174 tests pass. - Add E2EE upgrade/migration docs to Matrix user guide.
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…tion (NousResearch#7981) Fixes NousResearch#7952 — Matrix E2EE completely broken after mautrix migration. - Replace MemoryCryptoStore + pickle/HMAC persistence with mautrix's PgCryptoStore backed by SQLite via aiosqlite. Crypto state now persists reliably across restarts without fragile serialization. - Add handle_sync() call on initial sync response so to-device events (queued Megolm key shares) are dispatched to OlmMachine instead of being silently dropped. - Add _verify_device_keys_on_server() after loading crypto state. Detects missing keys (re-uploads), stale keys from migration (attempts re-upload), and corrupted state (refuses E2EE). - Add _CryptoStateStore adapter wrapping MemoryStateStore to satisfy mautrix crypto's StateStore interface (is_encrypted, get_encryption_info, find_shared_rooms). - Remove redundant share_keys() call from sync loop — OlmMachine already handles this via DEVICE_OTK_COUNT event handler. - Fix datetime vs float TypeError in session.py suspend_recently_active() that crashed gateway startup. - Add aiosqlite and asyncpg to [matrix] extra in pyproject.toml. - Update test mocks for PgCryptoStore/Database and add query_keys mock for key verification. 174 tests pass. - Add E2EE upgrade/migration docs to Matrix user guide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7952
Summary
Matrix E2EE was completely broken after the mautrix-python migration. The bot could connect but never decrypt messages. Root cause was 6 distinct bugs, all fixed here.
Bugs fixed
Bug 1 — Initial sync drops to-device events
matrix.py:425-444never calledhandle_sync()on the initial sync response. Megolm key shares queued while the bot was offline were silently dropped (delivered once by the homeserver, never re-sent).Bug 2 — Pickle-based crypto store is fragile
MemoryCryptoStore+ custom pickle/HMAC persistence tied the serialized format to the exact mautrix version, broke across library migrations (nio → mautrix), and lost all crypto state when the pickle was incompatible.Fix: Replaced with mautrix's
PgCryptoStorebacked by SQLite viaaiosqlite. This is the same store mautrix bridges use in production. Crypto state auto-persists on every write — no manual serialization.Bug 3 — No device key verification on startup
After restoring crypto state from disk, hermes never checked that the homeserver still had the correct device keys. A restored account with
shared=Truewhose server-side keys were lost would silently fail.Fix: Added
_verify_device_keys_on_server()— queries/keys/queryafterolm.load(). Handles: keys missing (re-uploads), stale keys from migration (attempts re-upload with old device deletion), corrupted state (refuses E2EE with actionable error).Bug 4 —
datetimevsfloatTypeError crashes startupsession.py:suspend_recently_active()comparedentry.updated_at(datetime) withtime.time() - max_age_seconds(float), raisingTypeError: '>=' not supported.Bug 5 — Redundant
share_keys()floods logsExplicit
share_keys()in the sync loop fired on every iteration. OlmMachine already handles this via theDEVICE_OTK_COUNTevent handler. The redundant call produced "No one-time keys nor device keys got" warnings 6+ times in 40 seconds.Bug 6 —
MemoryStateStoremissingfind_shared_rooms(found during E2E testing)OlmMachine's decryption path calls
find_shared_rooms()on the state store, butMemoryStateStoredoesn't implement this method. Added_CryptoStateStoreadapter that wrapsMemoryStateStoreand provides the required interface.Changes
gateway/platforms/matrix.pygateway/session.pypyproject.tomlaiosqlite>=0.20,asyncpg>=0.29to[matrix]extratests/gateway/test_matrix.pywebsite/docs/.../matrix.mdBreaking changes
Users upgrading from a previous version with
MATRIX_ENCRYPTION=trueneed to:~/.hermes/platforms/matrix/store/crypto_store.pickleandcrypto.db/discardsessionin Element to force a new Megolm sessionNew installations are unaffected.
Test plan