feat: config-dir lever, ndjson source adapter, stored metadata on hits - #2174
Draft
joshuafontany wants to merge 4 commits into
Draft
feat: config-dir lever, ndjson source adapter, stored metadata on hits#2174joshuafontany wants to merge 4 commits into
joshuafontany wants to merge 4 commits into
Conversation
Adds `--source <adapter-name>` to `mempalace mine`. When supplied, the adapter resolves through resolve_adapter_for_source(), runs against the existing PalaceContext, and files its DrawerRecords through the standard write path. Omitting it leaves the legacy --mode dispatch untouched. --mode and --source select the ingest path and cannot both apply, so they sit in a mutually exclusive group. --mode carries no argparse default: argparse counts an option as seen for exclusivity only when its value differs from its default, so a default of "projects" would admit `--source X --mode projects` while rejecting `--source X --mode convos`. cmd_mine restores "projects" when --mode goes unsupplied, so `mempalace mine <dir>` behaves as before. Tests cover registry resolution, PalaceContext construction, adapter metadata landing verbatim, unknown-adapter failure, dry-run, the exclusivity across all three mode spellings, and help documenting --source as the RFC 002 extension path. Closes MemPalace#2062 tests/test_mine_source_registry.py: 5 passed. mine/cli selection: 752 passed, 3 skipped.
…verride `MempalaceConfig.__init__` took `config_dir` as a constructor argument documented "useful for testing", with no environment or CLI lever — so the config directory hardwires to `~/.mempalace` for every spawned process, while `palace_path` already honors `MEMPALACE_PALACE_PATH`. A host that embeds mempalace as a sidecar spawns it as a process: it can pass env and argv, and nothing else. The config file supplies `backend`, `collection_name`, `embedding_model`, `write_routing`, `palace_path`, the milvus/qdrant/pgvector sets and ~20 more, so an embedding host inherits every one of them from a user-level file it does not own. Countering key-by-key with env vars does not scale to that surface; one symmetric lever closes it. Resolution order matches `palace_path`: explicit argument > environment > default. `~` expands, a blank value reads as unset, and an unset variable leaves `~/.mempalace` exactly where it stood — the change adds a lever and moves no existing behaviour. Six tests cover the promised order: env redirects both config-dir-derived files; a config.json under the env-named directory actually feeds resolved values; an explicit argument outranks the env; unset and blank both keep the home default; `~` expands. Revert-verified: with the change reverted, 3 of the 6 fail — the guard bites rather than passing vacuously. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NJGuNb4oxv4C8RJH8TJx
…ted records
A generic first-party source adapter (RFC 002 §3.2): each NDJSON line is one
pre-extracted record {content, source_file, metadata, chunk_index} filed as one
verbatim drawer. Byte-preserving (no declared transformations), per-source_file
chunk-index so records sharing a source_file never collide on the deterministic
drawer id, optional --wing routing fallback. Registered first-party in
sources/__init__.py (not the third-party entry-point group). 11 conformance tests.
Carries zero caller-specific vocabulary — any pipeline that pre-extracts records
can mine --source ndjson <spool>.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hits from search_memories() carried only display fields (text, wing, room, similarity, distances); every field a consumer stamped at ingest was dropped in flattening. Downstream filters and re-rankers (session-aware recall, provenance rules, custom pipelines) need those fields to match on. Each hit now carries its stored metadata dict whole. Covered by a seeded round-trip test; internal underscore-prefixed fields still strip as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
joshuafontany
force-pushed
the
feat/2110-config-dir-ndjson-hit-metadata
branch
from
August 8, 2026 19:43
3f8a0ab to
18a15cd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — the remaining three faces of #2110, held until #2173 lands.
Stacked on #2173. The ndjson adapter reaches a palace through
mine --source, so its end-to-end tests import_mine_via_source_adapterfrom that PR. The registry commit therefore rides at the base of this diff, and drops out once #2173 merges and this rebases ontodevelop.MEMPALACE_CONFIG_DIR—MempalaceConfigtakesconfig_diras a constructor argument with no environment lever, whilepalace_pathalready honorsMEMPALACE_PALACE_PATH. Every process on a machine therefore reads one~/.mempalace/config.json:backend,collection_name,embedding_model,write_routing, the milvus/qdrant/pgvector sets. This makes the two resolutions mirror.ndjson source adapter — an RFC 002 citizen for a host that already holds its records: a JSON-Lines spool of pre-extracted records, one per line, filed verbatim. Declares
byte_preserving,whole_record, no transformations, and registers in-tree the way RFC 002 reserves for first-party adapters.Stored metadata on search hits — a hit returns what the store already knew, so a caller reading a result does not re-derive fields the drawer carries. Complements the
source_filefiltering from #1815.Each of the three is independent and touches a disjoint file set, so any subset can be taken and the rest dropped.
Suite on the stack: ruff clean; ndjson / config-dir / searcher / source selection 179 passed.