layers: add read_diary public API - #3
Merged
Merged
Conversation
Replaces the fixed-width 800-char hard-cut chunker that produced mid-word fragments (59% of drawers in production palace) and indexed tool-output noise (logs, ps listings, line-numbered diffs, file listings, truncation messages) as memory. New chunker.py module provides: - smart_split(text, target, ceiling): boundary-aware splitting that scans backward from ceiling for paragraph -> sentence -> newline -> word boundaries; punctuation stays with the previous chunk. - is_excluded_content(text): line-ratio heuristics that drop chunks dominated by log lines, ps rows, line-numbered diff/grep output, arrow-redirected tool output, or standalone truncation markers. - Code-block atomicity: triple-backtick fences are never split mid- listing; oversized blocks emit as one atomic chunk rather than fragmenting. convo_miner._chunk_by_exchange now preserves the response's original newlines (paragraph and code-fence structure) instead of stripping each line and joining with single spaces. Real-data validation on production transcripts: mid-line starts 8% -> 0%, length-cliff at 800 chars 50% -> 0%, ~75% drawer reduction from excluding tool-output noise. 991 tests pass (+20 new chunker tests). Forward-only fix: existing 95K drawers are unchanged. Re-mining those source files would apply the new chunker; deferred pending recall-quality telemetry from new mining cycles.
Audit of the 7b27608 chunker fix found that mid-line drawers appeared on post-commit production data at 40% rather than the claimed 0%. Root cause: 96% of bad post-commit drawers came from a single tool-results .txt file under .claude/projects/, where Claude Code spills oversized tool outputs. That file contained a Python traceback with embedded Claude Code session JSONL inlined as the "filename" of an OSError. normalize() returned it verbatim because none of the JSON parsers extracted messages, then chunk_exchanges fell through to paragraph-mode and smart_split hard-cut at ceiling because the JSON blob has no natural-language boundaries. Two-layer fix: (1) Source-set hygiene — palace.SKIP_DIRS gains "tool-results". Claude Code per-session tool-results subdirectories contain raw tool artifacts (JSON dumps, log captures, error tracebacks with inlined session metadata), not conversation content. Skipping the subtree at the walker level prevents the failure mode at the source. (2) Defense-in-depth — chunker.is_excluded_content gains JSON-blob detection. Two new heuristics: - >=50% of non-empty lines start with `{"key":` (the canonical JSONL session-log shape) - High `,"key":` density (>=1 per 200 chars) combined with a transcript marker (uuid/sessionId/requestId/messageId/ parentUuid/timestamp:YYYY-) is decisive Real prose mentioning a uuid or showing one inline JSON example survives — only blob-level density triggers exclusion. Also: convo_miner._emit_chunks and miner.chunk_text now apply is_excluded_content per-chunk in addition to the whole-content check. A largely-prose response with one embedded log/JSON paragraph drops just that chunk rather than the whole exchange. Validation against real production sources: - tool-results/b5k5gj8gj.txt (the actual 82-bad-drawer culprit): whole-file excluded, 0 chunks produced (was 82 mid-line drawers) - real Claude Code .jsonl session: 79K -> 13K transcript via normalize, 11 prose chunks, 0% length-cliff @800 - prose with inline JSON example: kept as one clean chunk Tests: 997 pass (was 991), +5 chunker exclusion cases, +1 walker skip-dir case. Forward-only — existing 95K drawers untouched. The 82 noise drawers from the b5k5gj8gj.txt artifact remain in the palace pending a separate selective-delete decision.
Adds a clean public API for reading an agent's diary entries from the
palace, encapsulating the chromadb where-filter + sort + slice logic
that consumers (Vestige's runtime_orientation, primarily) were
inlining via mempalace.palace.get_collection.
Surface:
- DiaryEntry dataclass (frozen): date, filed_at, topic, content.
Mirrors the persistence shape used by tool_diary_write /
tool_diary_read in mcp_server.
- DiaryUnavailable exception: raised when the palace is unreachable
or the diary collection cannot be queried. Distinguishes
infrastructure failure (palace missing, chromadb error) from a
genuinely empty diary (returns []). Callers who care can render
differently: DiaryUnavailable -> "(diary unavailable)", [] ->
"(no entries yet)".
- read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry]:
Filters wing=wing_{agent.lower()} room=diary, sorts by filed_at
descending, slices to last_n.
Side-effect-free: uses palace.get_collection (not mcp_server, which
performs dup2(stderr, stdout) at import time as part of the MCP stdio
protocol contract and would clobber consumer log streams).
Consumer migration: Vestige's runtime_orientation moves from inline
chromadb logic to this API in a coordinated PR. The TODO at
runtime_orientation.py:196 was the load-bearing comment that named
this exact API; that comment goes away in the consumer migration.
Tests: 10 new in tests/test_read_diary.py covering happy-path sort/
last_n/empty/wing-filter/case-insensitive-agent, plus DiaryUnavailable
raised on get_collection failure + col.get failure, plus palace_path
override semantics.
10/10 pass. No regression to existing layers tests.
3 tasks
jpwinans
pushed a commit
that referenced
this pull request
Jul 3, 2026
Adds _try_gemini_json parser to normalize.py for three layouts:
1. Gemini API contents format (~/.gemini/sessions/*.json):
{"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}
2. Messages-wrapper variant:
{"messages": [{"role": "user", ...}, {"role": "model", ...}]}
3. Flat top-level list with role="model".
This complements the existing _try_gemini_jsonl parser (which handles
~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata
sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers
exported / Studio-saved transcripts.
## Review feedback addressed (PR MemPalace#204)
bgauryy review:
- #1 Parser-precedence bug: _try_gemini_json runs *before*
_try_claude_ai_json so the {"messages":[..., role=model, ...]}
layout is no longer silently claimed by the Claude parser. The
Gemini parser's has_model_role guard prevents false-positives
against Claude / ChatGPT data.
- #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format
+ test_messages_wrapper_does_not_get_claimed_by_claude pin the
fix in place.
- #3 Test conflicts with current main: rebased onto develop;
tests restructured into TestGeminiJson class.
- #4 tempfile/os.unlink → pytest tmp_path everywhere.
- #5 elif not text → else (the elif branch was dead).
- #6 Module docstring updated to mention Google AI Studio.
Tests: 9 new cases in TestGeminiJson covering all three layouts,
multi-part text joining, non-text part skipping, has_model_role
disambiguation, dispatch-chain regression for review #1.
jpwinans
pushed a commit
that referenced
this pull request
Jul 3, 2026
Five fixes from the Gemini Code Assist review on MemPalace#1632 — three real bugs, two cleanups, all consistent with the bash-3.2-compatibility contract documented in the original commit. Bug fixes (high) ---------------- 1. hooks/cursor/lib/common.sh — config.json kill-switch check used a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)` command substitution. The heredoc body contains parens which trips the macOS bash 3.2.57 parser bug. Replaced with a `python -c '...'` call passing the config path as argv[1]. Matches the pattern already used in mempal_parse_stdin in the same file. 2. hooks/cursor/install.sh — a relative `--install-dir` was written verbatim into hooks.json. Cursor invokes hook commands from its own working directory (typically the project root), so a relative command path would silently fail to launch the hook. Now resolved to an absolute path against `$PWD` before being baked in. 3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0` would crash bash on `$((NEXT % 0))` (division by zero). Extended the existing sanitiser case to coerce 0 to the default interval alongside empty / non-numeric values. Cleanups (medium) ----------------- 4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now inlined as `python -c '...'`. Removes a small leak window (tmpfile would linger if the script were interrupted between mktemp and rm -f) and shortens the script. 5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different semantics on BSD (macOS) vs GNU mktemp. Switched to the portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"` which behaves identically on both. Regression tests ---------------- - tests/test_cursor_hooks_shell.py test_save_interval_zero_is_coerced_to_default — guards fix #3. - tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath class: test_relative_install_dir_is_absolutized_in_hooks_json — guards fix #2 against regression. test_absolute_install_dir_is_preserved_verbatim — guards that the relative-to-absolute resolution does not mangle paths that were already absolute. Verification ------------ - bash -n on all three edited scripts: clean. - uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests). - uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed, 3 skipped (pre-existing). - uv run ruff check . / ruff format --check .: clean. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry]as a public API inmempalace.layers. Encapsulates the chromadb where-filter + sort + slice logic that consumers (Vestige'sruntime_orientation, primarily) were inlining viapalace.get_collection.Public surface
DiaryEntry(frozen dataclass) —date,filed_at,topic,content. Mirrors persistence shape oftool_diary_write/tool_diary_readinmcp_server.DiaryUnavailable(exception) — raised when palace is unreachable or chromadb query fails. Distinguishes infrastructure failure from genuinely empty diary (returns[]). Callers who care:DiaryUnavailable→"(diary unavailable)",[]→"(no entries yet)".read_diary— filterswing=wing_{agent.lower()} room=diary, sorts byfiled_atdescending, slices tolast_n.Why
Consumer (Vestige's
runtime_orientation) had a# TODO: migrate to mempalace.layers.read_diarycomment naming this exact API. Inlining the where-filter + metadata-key knowledge in every consumer means schema changes here cascade across consumers. Centralized inlayers.py, schema knowledge stays in the package that owns the schema.Safety
palace.get_collection(notmcp_server, which performsdup2(stderr, stdout)at import time as part of the MCP stdio protocol contract and would clobber consumer log streams).col.get(...). No writes, no HNSW touch (avoids the chromadb metadata-update pathology we hit on 2026-04-17).Tests
10 new in
tests/test_read_diary.py:filed_atdesclast_nlimit[][]last_n=0returns[]DiaryUnavailableraised onget_collectionfailureDiaryUnavailableraised oncol.getfailurepalace_pathkwarg overrides config10/10 pass. No regression to existing
layers.pytests.Consumer coordination
This PR is the upstream half of a coordinated change. The consumer migration (Vestige's
runtime_orientationcallingread_diaryinstead of inliningget_collection) ships in a separate Vestige PR — that PR depends on this one merging first so its import resolves againstmain. Merge order: mempalace first, then Vestige.Vestige PR: jpwinans/vestige feat/mempalace-read-diary-api (commit e698d51).