feat(gateway): X Chat (encrypted X DMs) platform plugin - #68930
Conversation
૮ >ﻌ< ა ci reviewran on 316bc84 — fix(xchat): address review — valid event fields, persisted c
|
alt-glitch
added
type/feature
381106d to
1819dc4
Compare
| # Fields we always request on the events endpoint — the decrypt path needs | ||
| # encoded_event; sender_id/conversation_id drive session routing. | ||
| _EVENT_FIELDS = ( | ||
| "conversation_id,created_at_msec,encoded_event,id,sender_id" |
There was a problem hiding this comment.
created_at_msec isn't a valid field here and the endpoint 400s on unknown fields, so every poll fails and the bot never receives anything. Valid fields are id, conversation_id, conversation_token, created_at, encoded_event, is_trusted, previous_id, sender_id, message_event_signature. created_at works, or drop it since nothing reads it.
There was a problem hiding this comment.
Fixed in 316bc84 — dropped created_at_msec and switched to the documented created_at; the valid field set is now documented next to _EVENT_FIELDS, and a test asserts every requested field is in the documented set so an undocumented field can't sneak back in.
| return | ||
| # Drop the oldest half. | ||
| items = sorted(self._seen_event_ids.items(), key=lambda kv: kv[1]) | ||
| for eid, _ in items[: len(items) // 2]: |
There was a problem hiding this comment.
This will re-reply to old messages. The timestamp is only written the first time an id is seen, and the poll refetches each conversation's latest 50 every time since there's no cursor. Past 5000 ids the prune evicts ids that are still in some quiet conversation's window, and next poll those come back as new. A per conversation cursor (last processed event id) fixes it.
There was a problem hiding this comment.
Fixed in 316bc84 — replaced the timestamp-dedup scheme with a persisted per-conversation cursor (last processed event id, ~/.hermes/xchat/cursors.json, advanced monotonically and written atomically). The dedup set is now only same-session echo suppression; pruning it can no longer cause re-replies. Covered by test_cursor_survives_restart_and_prune.
|
|
||
| async def _poll_conversation(self, conv_id: str) -> None: | ||
| assert self._api is not None and self._crypto is not None | ||
| page = await self._api.get_events(conv_id, max_results=50) |
There was a problem hiding this comment.
No cursor or pagination: a burst of more than 50 events between polls gets dropped, and _backlog_loaded is in memory so everything received while the gateway was down gets marked seen on restart. A persisted per conversation cursor fixes both, plus the dedup problem above.
There was a problem hiding this comment.
Fixed in 316bc84 — the poll now pages back (up to the hard cap) until it reaches the cursor, so a >50-event burst isn't dropped (test_burst_larger_than_one_page_is_paginated), and the cursor persists to disk so a restart processes messages received while the gateway was down instead of marking them seen (same restart test). _backlog_loaded is gone; the only remaining backlog-swallow is the very first sight of a conversation ever (no cursor on disk), which is the intended don't-reply-to-history behavior.
| canonical = chat_id | ||
| if events_b64: | ||
| try: | ||
| batch = crypto.decrypt_batch(events_b64) |
There was a problem hiding this comment.
set_signing_keys is never called before decrypt_batch here, so KeyChange verification can't pass and the conversation key never seeds. Standalone sends will fail against the real SDK. The gateway path does it correctly (_register_signing_keys first). Tests miss it because FakeCrypto.decrypt_batch hands out keys unconditionally.
There was a problem hiding this comment.
Fixed in 316bc84 — _standalone_send now fetches the participants' public keys and pushes them via set_signing_keys before decrypt_batch, mirroring the gateway path. Also added a StrictCrypto test fake that refuses to hand out conversation keys until set_signing_keys is called; it fails against the old code and passes now (test_standalone_send_registers_signing_keys_before_decrypt).
| except Exception as e: | ||
| logger.warning("[xchat] key-change processing failed conv=%s: %s", conv_id, e) | ||
| continue | ||
| if etype != "Message": |
There was a problem hiding this comment.
This skip drops edits. The feed sometimes returns an edited message only as the edit event and drops the original, so an edited message is invisible to the bot forever, including edits to a message it hasn't answered yet.
There was a problem hiding this comment.
Fixed in 316bc84 — MessageEdit events are now dispatched like messages (message_text accepts both), so an edit-only delivery is no longer invisible. Covered by test_message_edit_dispatched.
|
|
||
| crypto = XChatCrypto() | ||
| blob_path = _blob_path() | ||
| resuming = blob_path.exists() and marker.get("body") and not force |
There was a problem hiding this comment.
If the blob exists but registration.json is missing or corrupt, this falls through and regenerates over private_keys.b64. No forward secrecy in this protocol, so that permanently kills every existing conversation. Don't overwrite an existing blob without --force, or at least back it up first.
There was a problem hiding this comment.
Fixed in 316bc84 — with an existing blob and a missing/corrupt marker, register now refuses with an explanation instead of regenerating; --force (now accepted by register too) backs the old blob up to a timestamped .bak (mode 600) before minting a new identity. Covered by test_register_refuses_to_overwrite_orphan_blob and test_register_force_backs_up_existing_blob.
| payload = crypto.generate_and_register_payload() | ||
| body = payload["registration"] | ||
| version = payload["version"] | ||
| blob_path.write_text(payload["private_keys_b64"] + "\n", encoding="utf-8") |
There was a problem hiding this comment.
write_text then chmod leaves a window where the file is world readable, and the chmod failure is swallowed. Use os.open with 0o600. A compromised blob exposes all past and future messages, so XCHAT_PRIVATE_KEYS_B64 deserves a warning in the docs too.
There was a problem hiding this comment.
Fixed in 316bc84 — the blob is written via os.open(O_WRONLY|O_CREAT|O_TRUNC, 0o600) so it's 600 from the first byte, with a follow-up chmod for pre-existing looser files. XCHAT_PRIVATE_KEYS_B64 now carries a compromise warning in both the messaging docs and the env-var reference.
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_standalone_send_requires_token(monkeypatch): |
There was a problem hiding this comment.
api.py has no coverage at all (401 refresh, token rotation, 429), and _standalone_send only tests the two config error returns, which is how the signing keys bug got through. A fake that refuses to give out conversation keys until set_signing_keys is called would have caught it.
There was a problem hiding this comment.
Fixed in 316bc84 — added tests/plugins/platforms/xchat/test_xchat_api.py (401 reactive refresh + rotation persistence, single-retry guard, proactive expiry refresh, 429 with reset epoch, documented-fields invariant, path hyphenation) via httpx.MockTransport, plus the StrictCrypto fake exercising the full _standalone_send happy path and the no-roster failure path.
Connects the Hermes gateway to X's end-to-end encrypted direct messages via the official X Chat API. All plaintext stays local: inbound encoded_event blobs are decrypted with the Chat XDK (chatxdk) and outbound replies are encrypted + signed before they reach X. - plugins/platforms/xchat/: adapter (polling inbound, encrypted send, typing, group mention gating, allowlist/pairing, cron standalone sender), async httpx API client with OAuth2 refresh-token rotation, Chat XDK crypto wrapper, and a resume-safe 'hermes xchat setup' CLI (token -> user id -> keygen -> rate-limit-aware key registration) - tools/lazy_deps.py + pyproject.toml: chatxdk lazy-install entry (platform.xchat) + xchat extra for packagers - hermes_cli/main.py: resolve a deferred bundled platform's CLI subcommand when invoked as 'hermes <platform>' — also fixes 'hermes photon' being unreachable since the lazy-load perf change (#54448) - docs: messaging guide, env-var reference, sidebar, platform tables - tests: 24 offline unit tests (dispatch, dedup, backlog seeding, KeyChange handling, mention gating, registry parity, crypto wrapper)
1819dc4 to
0fb31e2
Compare
…dits, signing keys, key-blob safety Addresses santiagomed's review on #68930: - api.py: drop created_at_msec from chat_message_event.fields — the endpoint 400s on unknown fields, so every poll failed. Use the documented created_at instead, and document the valid field set. - adapter.py: replace the in-memory backlog/dedup scheme with a persisted per-conversation cursor (~/.hermes/xchat/cursors.json). Fixes three defects at once: re-replies to old messages after dedup prune eviction, dropped bursts >50 events between polls (the poll now pages back to the cursor), and messages received while the gateway was down being swallowed as backlog on restart. - adapter.py/crypto.py: dispatch MessageEdit events — the feed can return an edited message only as the edit event, which previously made it invisible to the bot forever. - adapter.py: _standalone_send now fetches participants' public keys and calls set_signing_keys BEFORE decrypt_batch, so KeyChange verification can pass and the conversation key actually seeds against the real SDK. - cli.py: refuse to regenerate over an existing private-key blob when the registration marker is missing/corrupt (no forward secrecy — overwriting permanently kills every conversation); --force now backs the old blob up first, and the register subcommand accepts --force too. - cli.py: write the key blob via os.open(..., 0o600) — no world- readable window between write_text and chmod. - docs: XCHAT_PRIVATE_KEYS_B64 compromise warning (messaging page + env-var reference); inbound section updated for cursors/edits. - tests: API-layer coverage (401 refresh + rotation persistence, single-retry guard, proactive expiry refresh, 429 reset, documented event fields, path hyphenation), a StrictCrypto fake that refuses keys until set_signing_keys is called (would have caught the standalone-send bug), cursor restart/burst/edit tests, and CLI blob-guard/0600 tests.
0fb31e2 to
316bc84
Compare
… new-conversation handshake, key-event meta, read receipts
Brings the X Chat adapter to parity with mature gateway platforms:
- Encrypted media, both directions. Inbound attachments are downloaded
(GET /2/chat/media/{conv}/{hash}), decrypted with the conversation key
for the EVENT's key version (post-rotation media stays readable),
size-capped, cached locally, and surfaced on MessageEvent
(media_urls/media_types + correct MessageType) so vision/file tools
see them. Outbound send_image/send_image_file/send_voice/send_video/
send_document encrypt with the latest conversation key
(encrypt_stream), upload via the 3-step chat-media flow
(initialize/append/finalize, base64 JSON segments), and attach by
media_hash_key. Standalone sends (cron/send_message_tool) carry
media_files the same way.
- Native threaded replies. A bounded per-conversation cache of decrypted
events lets send(reply_to=...) use encrypt_reply against the real
target event; unknown targets fall back to a plain send. Inbound
reply context (reply_to_message_id/text/author,
reply_to_is_own_message) now propagates on MessageEvent.
- New-conversation initiation. Standalone send to a bare numeric user id
performs the conversation-key handshake: fetch both parties' public
keys, verify each identity↔signing binding (verify_key_binding — a
substituted key must never receive the conversation key), wrap a
fresh key per participant (prepare_conversation_key_change), POST to
add-conversation-keys, then encrypt under the returned raw key.
- meta.conversation_key_events. The events endpoint returns KeyChange
events SEPARATELY in meta — previously they were never decrypted, so
conversations whose key changes fell outside the data array could
never seed a key. Both the poll loop and the standalone sender now
feed them through the batch decrypt path (after signing-key
registration) before processing messages.
- Read receipts (opt-in, XCHAT_SEND_READ_RECEIPTS, default off) via
POST /2/chat/conversations/{id}/read.
- Latest-key-version tracking per conversation for media encrypt and
correct key selection after rotations.
api.py: media_upload (chunked 3-step), media_download, mark_read.
crypto.py: encrypt_reply, encrypt_media/decrypt_media, verify_key_binding,
prepare_conversation_key_change (SDK->API body mapping incl.
action_signatures), attachments/explicit-key support on encrypt_text,
latest_key_version surfaced from decrypt_events, message_attachments,
detect_mime_type/detect_image_dimensions helpers.
Docs: media/replies/handshake/read-receipts documented; stale "text
only" / "reply flows only" limitations removed; capability row added to
the messaging comparison table; media.write scope noted in setup + docs.
Tests: 51 total — inbound attachment decrypt-and-cache, outbound
encrypt-upload-attach (+ no-key failure), threaded-reply cache hit and
fallback, meta key-event absorption order, read-receipt opt-in/default,
reply-context propagation, full handshake happy path (bindings verified,
key change POSTed, explicit key used), chunked upload reassembly,
media download, mark-read body.
3 participants
Summary
Hermes can now run as a bot on X Chat — X's end-to-end encrypted DMs — as a bundled platform plugin: inbound
encoded_eventblobs are decrypted locally with the official Chat XDK and replies are encrypted + signed before they ever reach X, so the platform only routes ciphertext.Also fixes a live CLI bug: since the platform lazy-load perf change (#54448), deferred bundled platform plugins never register their
hermes <platform>CLI subcommand —hermes photonerrored with "invalid choice". The parser now resolves the one matching deferred loader when the first positional matches a registered platform.Changes
plugins/platforms/xchat/adapter.py: gateway adapter — conversation polling with auto-discovery, backlog batch-decrypt to seed the verified conversation-key cache (without replying to old messages), KeyChange rotation handling, per-sender signing-key roster, dedup, group mention gating, scoped credential lock, encrypted send + typing,_standalone_sendfor out-of-process cron delivery, fullPlatformEntryparity (allowlist/allow-all envs, cron home channel, setup_fn, env enablement, platform hint)plugins/platforms/xchat/api.py: async httpx client for the X API v2 chat endpoints with OAuth2 refresh-token rotation (proactive near-expiry + reactive on 401; rotated tokens re-persisted to.env), 429 handling with reset-epoch backoffplugins/platforms/xchat/crypto.py: thin Chat XDK wrapper (keygen/export/import,set_identity/set_cache_keys/set_signing_keys, batch + single decrypt, encrypt) — lazy-installschatxdkat first useplugins/platforms/xchat/cli.py:hermes xchat setup|register|status— resume-safe key registration (blob + payload persisted before any network call; a 429 or interrupt resumes the same identity instead of burning the ~few-per-24h registration budget)hermes_cli/main.py: resolve a deferred bundled platform's CLI subcommand when invoked ashermes <platform>(fixeshermes photontoo)tools/lazy_deps.py+pyproject.toml+uv.lock:platform.xchatlazy-dep entry (chatxdk==0.4.1) +xchatextra for packagersuser-guide/messaging/xchat.md, env-var reference section, sidebar, messaging index tables, integrations listtests/plugins/platforms/xchat/test_xchat_adapter.py— 24 offline tests (no network, no native SDK)Validation
scripts/run_tests.sh tests/plugins/platforms/xchat/tests/gateway/test_platform_registry.py,tests/hermes_cli/test_plugins.py,test_plugin_cli_registration.py,test_startup_plugin_gating.py,test_packaging_metadata.py,test_lazy_deps.pychatxdk(keygen → export → fresh-session import → set_identity → encrypt/decrypt round-trip → encrypt_message body shape)load_gateway_config()→ adapter construction →get_connected_platforms())hermes xchat status+hermes photon statusthrough the realmain()argparse pathNotes / scope
media_hash_keyflow) and initiating brand-new conversations (conversation-key handshake) are documented as not wired yet.Infographic