feat: add mine-device command for machine-wide discovery - #51
Closed
aktasbatuhan wants to merge 1 commit into
Closed
feat: add mine-device command for machine-wide discovery#51aktasbatuhan wants to merge 1 commit into
aktasbatuhan wants to merge 1 commit into
Conversation
Adds a new `mempalace mine-device` command that scans the local machine and populates the palace with everything it finds — git repos, AI agent sessions, tools, shell profile, and cloud connections. Solves the cold start problem: new users can fill their palace in one command instead of manually mining each project directory. What it scans: - AI agent sessions (Claude Code, Cursor, Codex) with CLAUDE.md extraction - Git repositories (language, framework, and contributor detection) - Obsidian vaults - Installed language runtimes and package managers - Shell profile (top commands, dotfiles) - Cloud provider configurations - Machine identity Wings are auto-detected from GitHub org names. CLAUDE.md files from agent-configured projects are filed verbatim into wing_self. No new dependencies. Includes 11 tests covering chunking, wing inference, agent session detection, dry-run mode, and end-to-end git repo ingestion.
Contributor
|
Interesting idea — machine-wide discovery is a bold feature. At +910 lines this is too big to land right now though, and auto-scanning a user's entire machine raises privacy/security concerns we'd want to think through carefully. If you're interested in contributing, reviewing incoming PRs would be really valuable — and it'll help you understand the codebase patterns for a future focused PR. Thanks! |
Author
|
@bensig Thanks a lot for the feedback! I understand all the concerns you raised, and will try to contribute in different ways. |
igorls
pushed a commit
that referenced
this pull request
May 13, 2026
…E queries
mine_convos was calling file_already_mined() once per file inside the
main loop. On a 150k-drawer palace, each per-file query
(`collection.get(where={"source_file": X}, limit=1)`) costs ~2 seconds
because chromadb has to scan the metadata index. A 2000-transcript
directory took >1h of wall-clock just to decide every file should be
skipped — and pegged multiple cores doing so, starving the daemon's
other endpoints.
bulk_check_mined() already existed for exactly this anti-pattern (its
docstring says "Callers can check membership and compare mtimes locally
instead of issuing one ChromaDB query per file") but only the project
miner used it; the convo miner kept the slow per-file path.
This patch adds a third helper, prefetch_mined_set(), that mirrors
file_already_mined()'s version-gate semantics (the check_mtime=False
branch used by mine_convos) and returns a set[str] for O(1) lookups.
mine_convos now calls it once before the loop; the loop body becomes
a set-membership check.
Observed on a 172k-drawer palace probing 10 files:
before: 21.2s (2.12s/file)
after: single bulk pass should be 30–60s for the whole 172k scan,
then 2000 O(1) checks ≈ free
file_already_mined() is kept for callers that genuinely need the
per-file semantics (the post-lock race-check in _file_chunks_locked
at convo_miner.py:350 still uses it intentionally).
Surfaced via jphein/familiar.realm.watch foundation-rework debugging
on 2026-05-11. Originally filed as #51.
igorls
pushed a commit
that referenced
this pull request
Aug 11, 2026
dialect.py: all 11 text-mode open() calls (6 read, 5 write) omitted encoding=, so on a non-UTF-8-locale process (e.g. German Windows / cp1252) UTF-8-written JSON and AAAK text is decoded via the OS codepage, corrupting umlauts. config.py: 4 text opens lacked encoding= (config.json + people_map read paths, two writes); the other json.dump write paths already pinned UTF-8. Audit findings #51 (dialect.py:360) and #84 (config.py:377). Regression: tests/test_encoding_hardening.py forces cp1252 default open and asserts umlaut round-trips through from_config / config.json read / raw-UTF-8 skip_name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've been working on AI agent memory for a while. Last year I built and contributed to mem-agent-mcp, an Obsidian-style markdown memory system for AI agents. At Dria, I've been building proactive, self-evolving memory into our agents (Kai and Dash), where the agent maintains its own context about the team, the codebase, and past decisions without being asked.
When I saw MemPalace, the alignment between my work and MemPalace interested me. Your metaphor maps directly to what I've been building personally. I wanted to contribute my own usage here, so I built something I kept running into as a problem: the cold start.
You install MemPalace, and the palace is empty. You have to manually
initandmineeach project directory. But the signal is already on the machine: dozens of git repos, AI agent sessions with CLAUDE.md files, tool configurations, shell history. It just needs to be discovered.This PR adds a
mine-devicecommand that scans the whole machine in one pass:What it discovers:
How it maps to the palace:
add_drawerpattern with all required metadata fieldsTested on a real machine:
What's in the PR:
mempalace/device_miner.py(~700 lines)tests/test_device_miner.py(11 tests)mempalace/cli.py(addedmine-devicesubcommand)No new dependencies. All 20 tests pass (9 existing + 11 new). Follows the same conventions as
miner.pyandconvo_miner.py.