feat(integration): hermes backfill, install command, and docs - #1
Closed
raman325 wants to merge 13 commits into
Closed
feat(integration): hermes backfill, install command, and docs#1raman325 wants to merge 13 commits into
raman325 wants to merge 13 commits into
Conversation
raman325
changed the title
There was a problem hiding this comment.
Pull request overview
Adds the Hermes integration “second half” by shipping a one-command installer (mempalace hermes install), a historical-session backfill script, and end-user documentation so Hermes can discover MemPalace as a native memory provider.
Changes:
- Add
mempalace hermes installcommand helpers to install the Hermes plugin, update~/.hermes/config.yaml, and optionally run backfill. - Add Hermes backfill implementation + a comprehensive unit test suite for the install helpers and idempotent backfill behavior.
- Add Hermes integration documentation (top-level README pointer + integration README), and bump dev ruff version in
uv.lock.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Bumps dev ruff pin from 0.15.18 → 0.15.20. |
tests/test_hermes_install_cli.py |
New tests covering Hermes install helper behavior and backfill idempotency/parity. |
README.md |
Adds a top-level Hermes integration section and pointer to integration docs. |
mempalace/integrations/hermes/README.md |
New integration guide: install, hooks, tool surface, configuration, and rationale for ChromaBackend usage. |
mempalace/integrations/hermes/backfill.py |
New backfill CLI/script for mining Hermes session exports into the palace using the canonical convo exchange path. |
mempalace/integrations/hermes/__init__.py |
Updates wing-routing helper docstring to reflect shared implementation used by backfill. |
mempalace/cli.py |
Adds Hermes install helpers + hermes install subcommand wiring and YAML-based config editing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
raman325
force-pushed
the
feat/hermes-backfill-install
branch
from
July 2, 2026 19:04
385f9ee to
3c7cf91
Compare
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add _is_real_user_message, _turn_fingerprint_from_messages, and _segment_turns module-scope helpers to the Hermes provider. These helpers distinguish Anthropic tool-result user messages from real user turns, fingerprint turn anchors using make_turn_fingerprint, and segment raw message lists into (user, assistant, turn_fp) dicts for the dedup safety-net scan path in on_session_end/on_pre_compress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When `messages` is passed to `sync_turn`, compute `_turn_fingerprint_from_messages` and carry the result as `turn_fp` in the worker payload. `_file_turn` then writes `turn_fp` into the drawer's extra metadata so future session_end / pre_compress safety nets can correlate already-filed turns by fingerprint without re-filing duplicates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add _session_lineage to MempalaceProvider to record ancestor session ids whose drawers cover an inherited transcript. Grows on /branch (parent carries forward), cleared on /reset and /resume (stale lineage would false-dedup against unrelated drawers — failure direction is always duplicate, never lose), and preserved through rewinds (same conversation). Also applies four small review-feedback items: empty-list guard in _is_real_user_message test, fingerprint call-site consistency test, clarifying comment above the empty-anchor early return in _turn_fingerprint_from_messages, and a separator comment in _segment_turns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…press on_session_end and on_pre_compress now enqueue _file_missing_exchanges, which scans the session's existing drawers (plus /branch ancestors) and files only turns sync_turn missed. Coverage tracked by turn_fp fingerprint (survives representation differences) with exact-text fallback. Empty session_id or scan failure disables dedup — failure direction is always duplication, never data loss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d tests The dedup scan now tracks (turn_fp, text) pairs per physical drawer and every skip consumes one whole drawer atomically across both match indexes. Independent fp/text counters let a single drawer protect two window occurrences of the same turn (fp match leaving an orphaned text count, or vice versa) — under-filing, i.e. a lost turn occurrence. Also: document the lineage snapshot-at-enqueue invariant and the Hermes reason-string coupling; cover branch-without-parent lineage clearing; de-StopIteration the fingerprint consistency test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A greedy single-pass match let an earlier segment's fingerprint-donor arm steal the drawer a later segment would have exact-matched — re-filing the covered turn while silently losing the missed one (interrupted 'continue' followed by a retried 'continue' reproduced it). Matching now runs exact-pair matches for every segment first, then fingerprint groups, then text groups; when same-fingerprint occurrences outnumber their drawers we file them all — bounded duplication of a covered turn over any possible loss. Loss paths now log at warning like the duplication paths, and session_end uses a bounded blocking enqueue since the session offers no later retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fingerprint coverage cannot see occurrences compression removed from the window: a byte-identical prompt recurring after its synced occurrence was compressed away gets fp-skipped, and if the recurrence was interrupted its partial assistant output is not captured (the user's words are already stored). Undecidable with occurrence counting alone — Hermes message dicts carry no per-message identity — and dropping fp-skip would re-file every tool-shaped turn. Documented in _file_missing_exchanges and pinned by test so any future correlation improvement surfaces loudly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fety net Windows CI caught the safety net losing a turn: on_session_end spawned the wake-up-cache refresh (a ChromaDB reader) concurrently with the worker's safety-net writes, sqlite locked, and the upsert failed with only a warning. Two defenses: _file_turn retries 'database is locked' with backoff before giving up, and the wake-up refresh is routed through the worker queue BEHIND the safety-net task so the provider's own reader can never race its writes (direct-thread fallback only when the queue is full, where the retries absorb the race). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second half of the MemPalace#1684 split — stacks on the provider core branch. Review-driven changes vs the original branch: - backfill.file_exchange routes through convo_miner.file_conversation_exchange, the same canonical write path live _file_turn uses, so historical and live drawers carry identical routing, normalization, and metadata (and authored_at now reflects the session file's mtime instead of the backfill run time). - backfill.classify_wing delegates to the provider's _match_wing_by_keywords instead of maintaining a synced copy; the parity test now guards the delegation surviving path-based loading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exchange drawer ids are salted with filed_at, so re-filing a session mints fresh ids instead of overwriting — every re-run of 'mempalace hermes install' duplicated all previously backfilled exchanges. Guard each session file with palace.file_already_mined (extract_mode="exchange"), the same per-source idempotency check the convo miner uses. Also extracts the by-path backfill module loader in the install CLI tests into a shared helper for the new re-run regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
raman325
force-pushed
the
feat/hermes-backfill-install
branch
from
July 6, 2026 19:41
3c7cf91 to
6eefb85
Compare
Owner
Author
|
Moved upstream as MemPalace#1942 (rebased onto feat/hermes-filing-dedup, draft while the stack under it merges). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of the MemPalace#1684 split per @igorls — backfill, the
mempalace hermes installcommand, and docs. Stacks on MemPalace#1915 (feat/hermes-provider-core); only the last two commits are new here, and the base will be retargeted todeveloponce that one merges.What this adds
One command after
pip install mempalace:Files land in
~/.hermes/plugins/mempalace/(the canonical user-installed plugin directory Hermes'discover_memory_providers()scans) with aplugin.yamlmanifest.~/.hermes/config.yamlgetsmemory.provider: mempalace. The command pip-installs mempalace into the Hermes venv, and optionally backfills existing Hermes session exports into the palace (--skip-backfillto opt out,--yesfor scripted installs).Also:
mempalace/integrations/hermes/README.md(install, hooks, tool surface, and why the ChromaBackend path avoids dimension mismatches) and a top-level README pointer.pyyamlbecomes a runtime dependency for theconfig.yamledit.How the review blockers were addressed
backfill.file_exchangeroutes throughconvo_miner.file_conversation_exchange(), the exact write path the live provider's_file_turnuses, so historical and live drawers carry identical routing, normalization, and metadata.authored_atis stamped from the session file's mtime, so date filters reflect when the conversation happened, not when the backfill ran.backfill.classify_wingdelegates to the provider's_match_wing_by_keywordsinstead of maintaining a hand-synced copy; a parity test guards that the delegation survives thespec_from_file_locationloading mode the install flow uses.ChromaBackend.get_or_create_collection()(same construction path as the runtime provider), never a rawPersistentClient.filed_at, so re-filing mints fresh ids rather than overwriting. Each session file is guarded withpalace.file_already_mined(..., extract_mode="exchange")— the same per-source check the convo miner uses — so re-runningmempalace hermes installdoesn't duplicate previously backfilled exchanges. Regression test runs backfill twice and asserts the drawer count holds.Known trade-off
Live turns use
source_file="hermes-session:{id}"while backfill uses the real session file path, so a session that was partially live-filed and later backfilled would not cross-dedup. Backfill targets pre-install history, so overlap should be rare — but it's a deliberate trade-off, not an oversight. Happy to tighten this if you'd rather key both on one convention.Tests
24 tests in
tests/test_hermes_install_cli.py: the path-resolution helpers (--hermes-homeguardrails, venv interpreter discovery, palace path precedence), atomicconfig.yamlediting (including the scalar-memory:and malformed-YAML cases), wing-routing parity, and the backfill re-run guard.