Add .mempalaceignore support and expand SKIP_DIRS

[pintamanta]

Summary

• Adds .pixi, .conda, .virtualenvs to the hardcoded SKIP_DIRS set. Projects using Pixi currently get 20k+ stdlib files indexed, drowning out actual project content (~200 files becomes ~53k files).
• Introduces .mempalaceignore — a project-root file using .gitignore syntax, applied on top of .gitignore rules during scan_project(). Lets users exclude paths tracked by git but not wanted in the palace (data dirs, static builds, media assets).

Motivation

Pixi manages Python environments inside the project (.pixi/envs/default/lib/python3.x/...), like node_modules. Without .pixi in SKIP_DIRS, every mine run indexes thousands of irrelevant stdlib files (ssl.py, typing.py, datetime.py), making the palace noisy and search useless.

The .mempalaceignore solves a related problem: directories tracked by git that should not be in the palace (e.g. data/, site/, binary asset folders). Today the only workaround is patching miner.py directly.

Changes

• miner.py: Added .pixi, .conda, .virtualenvs to SKIP_DIRS
• miner.py: New load_mempalaceignore_matcher() reusing GitignoreMatcher
• miner.py: scan_project() now loads and applies .mempalaceignore rules to dirs and files

Test plan

• Verify mine --dry-run on a project with .pixi/ skips the directory
• Create .mempalaceignore with data/ and *.png, verify exclusion
• Verify negation patterns work
• Verify no regression on projects without .mempalaceignore

[bgauryy]

PR Review: Add .mempalaceignore support and expand SKIP_DIRS

Executive Summary

Aspect	Value
PR	#379
Author	pintamanta
Target	main
Files Changed	1 (mempalace/miner.py, +71/-0)
Review Mode	Full
Risk	HIGH
Recommendation	Request Changes

This PR adds .mempalaceignore support (a project-root file using .gitignore syntax for excluding paths from mining) and expands SKIP_DIRS with .pixi, .conda, .virtualenvs. The motivation is sound — Pixi environments can inject 20k+ stdlib files into the index.

However, the PR is based on a stale fork that diverges significantly from current main. The fork's copy of miner.py predates a refactor that moved SKIP_DIRS, get_collection, and file_already_mined into palace.py and added a MAX_FILE_SIZE safety guard. Merging as-is will produce conflicts and risks silently losing the 10MB file-size cap.

---

Ratings

Dimension	Score	Notes
Correctness	4/10	Stale base will cause merge conflicts; parsing inconsistency in .mempalaceignore; missing MAX_FILE_SIZE guard
Security	6/10	No direct vulnerabilities, but losing MAX_FILE_SIZE opens DoS via giant files
Performance	8/10	New ignore matcher is lightweight; SKIP_DIRS additions are efficient set lookups
Maintainability	5/10	Duplicated parsing logic; no tests; SKIP_DIRS dual-defined in fork vs palace.py

---

Issues

HIGH — Stale Base: MAX_FILE_SIZE Guard Will Be Lost

Confidence: HIGH
Location: mempalace/miner.py (fork vs main HEAD)

Current main has a 10MB file-size safety cap:

MAX_FILE_SIZE = 10 * 1024 * 1024
# ...
if filepath.stat().st_size > MAX_FILE_SIZE:
    continue

The fork was branched before this was added. When the PR is rebased/merged, this guard is at risk of being silently dropped if conflict resolution goes wrong. Without it, a single multi-GB log or binary file would be chunked and embedded into ChromaDB, potentially exhausting memory and disk.

Fix: Rebase the PR branch onto current main. Verify MAX_FILE_SIZE is preserved after rebase.

---

HIGH — Stale Base: SKIP_DIRS Dual Definition Conflict

Confidence: HIGH
Location: mempalace/miner.py:19 (main) vs mempalace/miner.py:42-67 (fork)

On current main, SKIP_DIRS is imported from palace.py:

from .palace import SKIP_DIRS, get_collection, file_already_mined

The fork defines SKIP_DIRS inline in miner.py and removes the palace import entirely. After rebase, the three new entries (.pixi, .conda, .virtualenvs) must be added to palace.py's SKIP_DIRS — not re-declared locally — to keep a single source of truth.

Fix: After rebasing, add the three new entries to SKIP_DIRS in palace.py and keep the existing import in miner.py.

---

MEDIUM — Inconsistent .gitignore Escape Handling in load_mempalaceignore_matcher

Confidence: HIGH
Location: mempalace/miner.py — load_mempalaceignore_matcher function (fork line ~220-270)

The existing GitignoreMatcher.from_dir() handles escaped comment/negation characters:

if line.startswith("\\#") or line.startswith("\\!"):
    line = line[1:]
elif line.startswith("#"):
    continue

The new load_mempalaceignore_matcher skips this entirely:

if not line or line.startswith("#"):
    continue

A .mempalaceignore file containing \#not-a-comment would be incorrectly treated as a comment and silently ignored.

Fix: Either (a) extract a shared _parse_ignore_rules(lines) helper used by both from_dir() and load_mempalaceignore_matcher, or (b) add the \\#/\\! handling to the new function. Option (a) is preferred for DRY.

---

MEDIUM — .mempalaceignore File Not Added to SKIP_FILENAMES

Confidence: HIGH
Location: mempalace/miner.py — SKIP_FILENAMES set

The .gitignore file is in SKIP_FILENAMES to prevent it from being indexed:

SKIP_FILENAMES = {
    "mempalace.yaml",
    # ...
    ".gitignore",
    "package-lock.json",
}

The new .mempalaceignore file is not added here. If a user creates one, it will be mined and embedded as content — metadata leaking into the palace.

Fix: Add ".mempalaceignore" to SKIP_FILENAMES.

---

MEDIUM — No Tests for New Feature

Confidence: HIGH

The PR introduces a new user-facing feature (.mempalaceignore) and modifies the core scanning pipeline but includes zero tests. The existing test suite has tests/test_miner.py which should be extended. Key test cases needed:

1. .mempalaceignore file excludes matched paths
2. .mempalaceignore negation patterns (!) work correctly
3. Force-include paths override .mempalaceignore exclusions
4. Missing .mempalaceignore file = no-op (already handled, but should be tested)
5. New SKIP_DIRS entries (.pixi, .conda, .virtualenvs) are skipped

---

LOW — Duplicated Rule-Parsing Logic

Confidence: MED

load_mempalaceignore_matcher reimplements ~30 lines of rule parsing that GitignoreMatcher.from_dir() already does. The only difference is the source file (.mempalaceignore vs .gitignore). A cleaner approach:

@classmethod
def from_file(cls, file_path: Path, base_dir: Path = None):
    """Parse any .gitignore-syntax file into a matcher."""
    base_dir = base_dir or file_path.parent
    # ... shared parsing logic ...

Then from_dir and load_mempalaceignore_matcher both delegate to from_file.

---

Flow Impact Analysis

Changed Symbol	Blast Radius	Risk
SKIP_DIRS expansion	should_skip_dir() → scan_project() → all mine commands	LOW — additive, narrows scope
load_mempalaceignore_matcher (new)	Called once per scan_project() invocation	LOW — new leaf function
.mempalaceignore check in scan_project	All file mining operations	MED — new filter in hot path, but force-include override is preserved

---

Summary

Severity	Count	Key Items
HIGH	2	Stale base risks losing MAX_FILE_SIZE; SKIP_DIRS dual definition will conflict
MEDIUM	3	Incomplete escape handling; .mempalaceignore not in SKIP_FILENAMES; no tests
LOW	1	Duplicated parsing logic

Action Required: Rebase onto current main, add the three SKIP_DIRS entries to palace.py (not miner.py), fix the escape handling inconsistency, add .mempalaceignore to SKIP_FILENAMES, and add basic test coverage.

[web3guru888]

We run a multi-domain palace (5 wings, 208 discoveries) and .mempalaceignore is something we've wanted. This is a direct match for our use case — we have data directories tracked by git that shouldn't be indexed.

Strengths:

• Reusing GitignoreMatcher for .mempalaceignore is clean — same syntax, same matching behavior, no new parser to maintain.
• Adding .pixi, .conda, .virtualenvs to SKIP_DIRS is well-motivated. Pixi's in-project .pixi/envs/ is exactly like node_modules and should be skipped by default.
• The matcher loads once at the project root and applies to both dirs and files in scan_project().

A few things to consider:

1. No tests — The test plan is all manual checkboxes. This really needs at least one automated test for .mempalaceignore behavior (create a tmpdir with a .mempalaceignore, add some files, run scan_project(), assert exclusions). The test_miner.py file has good patterns for this — see test_scan_project_respects_gitignore for reference.

2. Negation pattern interaction — Your parser handles ! patterns, but is_gitignored() applies matchers in ancestor order (last match wins). Since .mempalaceignore is applied after .gitignore matchers, a ! in .mempalaceignore could un-ignore something .gitignore excluded — is that intentional? Might be worth documenting.

3. Heads-up: overlap with #403 — roip's Windows compat PR introduces .mpignore with the same concept (project-level ignore on top of .gitignore), but using a different filename. If both land, the project will have two competing ignore files (.mempalaceignore vs .mpignore). Worth coordinating on which name wins. I'd slightly prefer .mempalaceignore for explicitness, but .mpignore is shorter to type.

4. load_mempalaceignore_matcher only checks project root — Unlike .gitignore which can exist at any directory level, your implementation only loads from the project root. That's probably fine for v1 but worth noting in a docstring.

Would love to see this merged (with tests) — it unblocks a real workflow for multi-purpose repos.

🔭 Reviewed as part of the MemPalace-AGI integration project — autonomous research with perfect memory. Community interaction updates are posted regularly on the dashboard.

[mvalentsev]

Heads up -- #154 targets the same feature (#102) and already has a review. The implementation might differ, worth comparing before duplicating effort.

[bensig]

closing in favor of #154 which covers the same .mempalaceignore support. thanks @pintamanta!