Skip to content

Add Rust, Swift, and Xcode build dirs to SKIP_DIRS - #65

Open
shrhoads wants to merge 3 commits into
MemPalace:developfrom
shrhoads:add-skip-dirs-rust-swift
Open

Add Rust, Swift, and Xcode build dirs to SKIP_DIRS#65
shrhoads wants to merge 3 commits into
MemPalace:developfrom
shrhoads:add-skip-dirs-rust-swift

Conversation

@shrhoads

@shrhoads shrhoads commented Apr 7, 2026

Copy link
Copy Markdown

Summary

  • Adds target (Rust/Cargo), Pods (CocoaPods), .build (Swift PM), and DerivedData (Xcode) to all SKIP_DIRS sets
  • Updates all 5 definitions across miner.py, convo_miner.py, entity_detector.py, and room_detector_local.py

Problem

Mining a project containing a Rust target/ directory (commonly 5+ GB of compiled artifacts and intermediate build files) causes:

  1. Extreme memory usage (7+ GB RAM)
  2. A bloated HNSW link_lists.bin index (500+ GB in my case)
  3. Eventual segfault (exit code 139) when trying to use the palace afterward

The existing SKIP_DIRS covers JS/Python/Next.js build dirs but misses common compiled-language equivalents.

Test plan

  • Verified mempalace mine . --dry-run scans 346 files instead of tens of thousands after this change
  • Confirmed no regressions — non-build-artifact files are still scanned normally

🤖 Generated with Claude Code

Add `target` (Rust/Cargo), `Pods` (CocoaPods), `.build` (Swift PM),
and `DerivedData` (Xcode) to all SKIP_DIRS sets across miner,
convo_miner, entity_detector, and room_detector_local.

Without these, mining a project with a Rust `target/` directory
(often 5+ GB of compiled artifacts) causes extreme memory usage,
bloated HNSW indexes, and eventual segfaults.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@shrhoads

shrhoads commented Apr 7, 2026

Copy link
Copy Markdown
Author

Probably best long term fix is to add custom exclude dirs in yaml settings.

@bensig

bensig commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

Tests pass but lint fails — the long SKIP_DIRS line in room_detector_local.py needs formatting. A quick ruff format . should fix it.

@web3guru888 web3guru888 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👀 Review of #65Add Rust, Swift, and Xcode build dirs to SKIP_DIRS

Scope: +17/−1 · 4 file(s)

  • mempalace/convo_miner.py (modified: +4/−0)
  • mempalace/entity_detector.py (modified: +4/−0)
  • mempalace/miner.py (modified: +4/−0)
  • mempalace/room_detector_local.py (modified: +5/−1)

🟢 Approved — clean, well-structured PR. Good work @shrhoads!


🏛️ Reviewed by MemPalace-AGI · Autonomous research system with perfect memory · Showcase: Truth Palace of Atlantis

@bensig
bensig changed the base branch from main to develop April 11, 2026 22:23
@igorls igorls added enhancement New feature or request area/mining File and conversation mining labels Apr 14, 2026

@shrhoads shrhoads left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved merge from develop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/mining File and conversation mining enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants