feat: add --exclude flag and config-based exclude for mining

[adv3nt3]

Summary

Add user-configurable path exclusions for the mining pipeline. Currently, the only ways to skip paths are the hardcoded SKIP_DIRS set and .gitignore rules. This adds explicit --exclude support via both CLI flag and mempalace.yaml config.

Motivation

Projects often contain directories that are technically not gitignored but shouldn't be mined — test fixtures with hundreds of JSON files, vendored dependencies, generated code, large asset folders, or CI output directories. Today the only options are:

1. Add them to .gitignore — but they might be tracked intentionally (test data, vendored libs)
2. Hope SKIP_DIRS covers them — but it can't know every project's layout

Users setting up MemPalace for the first time shouldn't have to modify their .gitignore or know about hardcoded skip lists to get a clean mine. The --exclude flag and mempalace.yaml exclude: key give explicit, project-specific control without touching anything else in the repo.

Real examples:

• node_modules is in SKIP_DIRS, but vendor/, fixtures/, testdata/, __mocks__/ are not
• A project with tests/snapshots/ containing 500 JSON snapshot files — valid test data, tracked in git, but useless for memory mining
• generated/ or proto/ directories with auto-generated code that adds noise to the palace without adding context

Usage

CLI flag

# Exclude specific directories
mempalace mine ~/myproject --exclude dist,vendor,tmp

# Multiple --exclude flags
mempalace mine ~/myproject --exclude dist --exclude vendor --exclude generated

Config (mempalace.yaml)

wing: myproject
rooms:
  - name: backend
    description: Server code
    keywords: [backend]
  - name: general
    description: All other files
    keywords: []
exclude:
  - dist
  - vendor
  - tmp
  - src/generated

During init

  Exclude folders from mining?
  These paths will be skipped in addition to .gitignore and built-in skips.
  You can also add/edit these later in mempalace.yaml under 'exclude:'.
  Example: dist, vendor, tmp, generated

  Paths to exclude (comma-separated, or enter to skip): dist, vendor

How it works

Precedence (highest to lowest)

1. --include-ignored — force-includes always win (overrides all exclusions)
2. --exclude + config exclude — merged at runtime (union)
3. .gitignore — respected by default
4. SKIP_DIRS — hardcoded baseline (.git, node_modules, __pycache__, etc.)

Path matching

• Paths are project-relative, normalized to POSIX format
• exclude: [dist] → skips dist/ and everything under it
• exclude: [src/generated] → skips only src/generated/, not src/ itself
• Directory pruning happens during os.walk for efficiency — excluded subtrees are never traversed

Config merge

CLI --exclude and mempalace.yaml exclude are combined. This means you can set permanent exclusions in config and add one-off exclusions via CLI:

# Config already excludes dist and vendor; also exclude tmp for this run
mempalace mine ~/myproject --exclude tmp

Changes

3 files changed, 72 insertions, 2 deletions:

File	What
miner.py	Add is_excluded() helper, exclude_paths param to scan_project() + mine(), read config excludes, print in status
cli.py	Add --exclude argument (same pattern as --include-ignored), parse and pass to mine()
room_detector_local.py	Add get_exclude_paths() interactive prompt, update save_config() to write exclude list to YAML

What's NOT changed

• Convo mining (mine --mode convos): --exclude only applies to project mining. Convo mining has its own scan_convos() with separate SKIP_DIRS. Adding exclude support there is a natural follow-up.
• --yes (non-interactive) init: Defaults to empty exclude list — no change to automated flows.
• SKIP_DIRS: The hardcoded set is unchanged. --exclude is additive on top of it.

Test plan

• ruff check passes clean on all 3 files
• ruff format --check already formatted
• python3 -m py_compile compiles OK for all 3 files
• Pyright reports 0 new diagnostics (all pre-existing)
• --include-ignored override verified: force-included paths bypass exclude filtering
• Path normalization handles leading/trailing slashes

[adv3nt3]

Context on why I built this: I'm mining a 7.5k file repo that's been running for 14 hours now — most of that time spent on hundreds of JSON test fixture files that are necessary for the test suite and tracked in git, but completely useless as memory material. Can't gitignore them, can't remove them. With --exclude tests/fixtures this would have been a 20-minute run.

[adv3nt3]

@bensig Note: this is complementary to .gitignore support from PR #78, not a duplicate. Issue #56 asked for an --exclude flag or config-based exclusion list — not .gitignore support. .gitignore skips untracked files, but --exclude skips tracked files that shouldn't be mined (test fixtures, vendored dependencies, generated code). These are files you want in git but don't want in your palace.

[adv3nt3]

this is the conversation with Claude, btw:

Claude:
As for the current mine — let it finish. You're at ~6917/7530, only ~600 files left. Canceling now would waste the hours already spent. The data will still be usable even with the bloated test fixtures indexed.
After it completes:

pip install --upgrade mempalace — get the .gitignore support
mempalace mine ~/IdeaProjects/monorepo --mode projects — re-mine, this time it'll skip gitignored files and be much faster (incremental + fewer files)
The re-mine with .gitignore respect should clean up the index by not re-indexing those huge JSONs.

me:
but we don't have files in gitignore, how it will help?

Claude:
You're right — those JSON test fixtures are tracked by git, not gitignored. They're legitimate test data committed to the repo. So .gitignore support won't help here.

The options are:

• Add a .mempalaceignore file (if mempalace supports it — let me check the PR more carefully)
• Add those paths to .gitignore — but that would affect git tracking, which you probably don't want
• Live with the current index — the bloat is in storage/indexing time, not in search quality. ChromaDB embedding search will still return relevant results ranked by similarity. The JSONs will just sit there unused unless you search for something that matches it's content.

Let me check if the merged PR added any non-gitignore exclusion mechanism:
Fetch(https://github.com/milla-jovovich/mempalace/pull/78/files)
⎿ Received 514.7KB (200 OK)

Confirmed — no custom exclusion support. Only .gitignore and a hardcoded skip list.
So your real options are:

1. Do nothing — let the current mine finish. The JSONs waste disk space in ChromaDB but won't pollute search results (embedding similarity will rank actual code/docs higher).
2. Add a nested .gitignore in the test fixtures directory — this won't untrack already-tracked files from git, but mempalace's new code walks .gitignore during mining. However, since the files are already tracked, git won't actually ignore them
3. Open an issue/PR on mempalace for --exclude glob support or .mempalaceignore — that's the proper fix.

My recommendation: Let the current mine finish (you're ~600 files away). The index works fine with the extra data — it just took longer to build. For conversations mining next, there won't be this problem since those files are small. If the index size or search noise bothers you later, open an issue on the repo for --exclude support.

[adv3nt3]

24h+: [6977/7530] and it's still running. we need this exclude setup

[web3guru888]

✨ Review of #161 — feat: add --exclude flag and config-based exclude for mining

Scope: +79/−2 · 4 file(s)

• mempalace/cli.py (modified: +10/−0)
• mempalace/miner.py (modified: +42/−0)
• mempalace/room_detector_local.py (modified: +23/−2)
• tests/test_cli.py (modified: +4/−0)

Strengths

• ✅ Includes test coverage

🟢 Approved — clean, well-structured PR. Good work @adv3nt3!

---

_{🏛️ Reviewed by MemPalace-AGI · Autonomous research system with perfect memory · Showcase: Truth Palace of Atlantis}

[igorls]

Hi, thanks for the contribution.

This PR has merge conflicts with develop, and the branch has not been updated in over 7 days, which puts it before our most recent release. The conflicts are likely against work that landed in that release.

Could you rebase onto develop so we can take another look?

If this change is no longer relevant, feel free to close the PR.

(This message is part of a periodic backlog pass, sent to all open PRs that match this state.)

[adv3nt3]

@igorls rebased onto develop. Conflicts: mine() and _mine_impl() both grew a new files: list = None parameter on develop (for the init-side pre-scanned-list optimization), and cmd_mine grew a _run_pass_zero redetect-origin path. Merged exclude_paths cleanly with both — kept the new if files is None: guard, threaded exclude_paths through to _mine_impl, and left _run_pass_zero untouched. Full tests/test_miner.py + tests/test_cli.py (96 tests) pass + ruff clean.