-
Notifications
You must be signed in to change notification settings - Fork 7.6k
feat: new MCP tools — get/list/update drawer, hook settings, export #635
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
ecae3b8
feat: MCP reliability — inode detection, WAL rotation, metadata cache…
jphein 2a41844
feat: new MCP tools (get/list/update drawer, hook settings, memories …
jphein a88de9e
fix: normalize.py handles string messages and Read offset type mismatch
jphein 49e9c64
fix: params null guard, L2→cosine docs, empty tool_use_map key guard
jphein 248555d
fix: rename ambiguous var 'l' to 'line' (E741 lint)
jphein File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| """ | ||
| exporter.py — Export the palace as a browsable folder of markdown files. | ||
|
|
||
| Produces: | ||
| output_dir/ | ||
| index.md — table of contents | ||
| wing_name/ | ||
| room_name.md — one file per room, drawers as sections | ||
|
|
||
| Streams drawers in paginated batches so memory usage stays bounded | ||
| regardless of palace size. | ||
| """ | ||
|
|
||
| import os | ||
| import re | ||
| from collections import defaultdict | ||
| from datetime import datetime | ||
|
|
||
| from .palace import get_collection | ||
|
|
||
|
|
||
| def _safe_path_component(name: str) -> str: | ||
| """Sanitize a string for use as a directory/file name component.""" | ||
| name = re.sub(r'[/\\:*?"<>|]', '_', name) | ||
| name = name.strip('. ') | ||
| return name or 'unknown' | ||
|
|
||
|
|
||
| def export_palace(palace_path: str, output_dir: str, format: str = "markdown") -> dict: | ||
| """Export all palace drawers as markdown files organized by wing/room. | ||
|
|
||
| Streams drawers in batches of 1000 and writes each wing/room file | ||
| incrementally, keeping memory usage proportional to batch size rather | ||
| than total palace size. | ||
|
|
||
| Args: | ||
| palace_path: Path to the ChromaDB palace directory. | ||
| output_dir: Where to write the exported markdown tree. | ||
| format: Output format (currently only "markdown"). | ||
|
|
||
| Returns: | ||
| Stats dict: {"wings": N, "rooms": N, "drawers": N} | ||
| """ | ||
| col = get_collection(palace_path) | ||
| total = col.count() | ||
|
|
||
| if total == 0: | ||
| print(" Palace is empty — nothing to export.") | ||
| return {"wings": 0, "rooms": 0, "drawers": 0} | ||
|
|
||
| os.makedirs(output_dir, exist_ok=True) | ||
|
|
||
| # Track which room files have been opened (so we can append vs overwrite) | ||
| opened_rooms: set[tuple[str, str]] = set() | ||
| # Track stats per wing: {wing: {room: count}} | ||
| wing_stats: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int)) | ||
| total_drawers = 0 | ||
|
|
||
| print(f" Streaming {total} drawers...") | ||
| offset = 0 | ||
| while offset < total: | ||
| batch = col.get(limit=1000, offset=offset, include=["documents", "metadatas"]) | ||
| if not batch["ids"]: | ||
| break | ||
|
|
||
| # Group this batch by wing/room so we do one file write per room per batch | ||
| batch_grouped: dict[str, dict[str, list]] = defaultdict(lambda: defaultdict(list)) | ||
| for doc_id, doc, meta in zip(batch["ids"], batch["documents"], batch["metadatas"]): | ||
| wing = meta.get("wing", "unknown") | ||
| room = meta.get("room", "general") | ||
| batch_grouped[wing][room].append({ | ||
| "id": doc_id, | ||
| "content": doc, | ||
| "source": meta.get("source_file", ""), | ||
| "filed_at": meta.get("filed_at", ""), | ||
| "added_by": meta.get("added_by", ""), | ||
| }) | ||
|
|
||
| # Write/append each room file | ||
| for wing, rooms in batch_grouped.items(): | ||
| safe_wing = _safe_path_component(wing) | ||
| wing_dir = os.path.join(output_dir, safe_wing) | ||
| os.makedirs(wing_dir, exist_ok=True) | ||
|
|
||
| for room, drawers in rooms.items(): | ||
| safe_room = _safe_path_component(room) | ||
| room_path = os.path.join(wing_dir, f"{safe_room}.md") | ||
| key = (wing, room) | ||
| is_new = key not in opened_rooms | ||
|
|
||
| with open(room_path, "a" if not is_new else "w", encoding="utf-8") as f: | ||
| if is_new: | ||
| f.write(f"# {wing} / {room}\n\n") | ||
| opened_rooms.add(key) | ||
|
|
||
| for drawer in drawers: | ||
| source = drawer["source"] or "unknown" | ||
| filed = drawer["filed_at"] or "unknown" | ||
| added_by = drawer["added_by"] or "unknown" | ||
|
|
||
| f.write( | ||
| f"## {drawer['id']}\n" | ||
| f"\n" | ||
| f"> {_quote_content(drawer['content'])}\n" | ||
| f"\n" | ||
| f"| Field | Value |\n" | ||
| f"|-------|-------|\n" | ||
| f"| Source | {source} |\n" | ||
| f"| Filed | {filed} |\n" | ||
| f"| Added by | {added_by} |\n" | ||
| f"\n" | ||
| f"---\n\n" | ||
| ) | ||
|
|
||
| wing_stats[wing][room] += len(drawers) | ||
| total_drawers += len(drawers) | ||
|
|
||
| offset += len(batch["ids"]) | ||
|
|
||
| # Build and print stats | ||
| index_rows = [] | ||
| for wing in sorted(wing_stats): | ||
| rooms = wing_stats[wing] | ||
| wing_drawer_count = sum(rooms.values()) | ||
| index_rows.append((wing, len(rooms), wing_drawer_count)) | ||
| print(f" {wing}: {len(rooms)} rooms, {wing_drawer_count} drawers") | ||
|
|
||
| # Write index.md | ||
| today = datetime.now().strftime("%Y-%m-%d") | ||
| index_lines = [ | ||
| f"# Palace Export — {today}\n", | ||
| "", | ||
| "| Wing | Rooms | Drawers |", | ||
| "|------|-------|---------|", | ||
| ] | ||
| for wing, room_count, drawer_count in index_rows: | ||
| index_lines.append(f"| [{wing}]({wing}/) | {room_count} | {drawer_count} |") | ||
| index_lines.append("") | ||
|
|
||
| index_path = os.path.join(output_dir, "index.md") | ||
| with open(index_path, "w", encoding="utf-8") as f: | ||
| f.write("\n".join(index_lines)) | ||
|
|
||
| stats = {"wings": len(wing_stats), "rooms": sum(r for _, r, _ in index_rows), "drawers": total_drawers} | ||
| print(f"\n Exported {stats['drawers']} drawers across {stats['wings']} wings, {stats['rooms']} rooms") | ||
| print(f" Output: {output_dir}") | ||
| return stats | ||
|
|
||
|
|
||
| def _quote_content(text: str) -> str: | ||
| """Format content for a markdown blockquote, handling multiline.""" | ||
| lines = text.rstrip("\n").split("\n") | ||
| return "\n> ".join(lines) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
set_hook_setting()writes directly toself._config_filebut doesn’t ensureself._config_direxists. On a first-run system (no~/.mempalace/yet), the write will fail and be silently swallowed, somempalace_hook_settingscan report success without persisting changes. Ensure the config directory exists (mkdir) and consider surfacing write failures to the caller.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicate of the same comment on PR #633 — will fix.