Skip to content

feat: implement bloom-filter-v3 and git-miner - #1418

Open
daedalus wants to merge 6 commits into
MemPalace:developfrom
daedalus:feature/implement-specs
Open

daedalus wants to merge 6 commits into
MemPalace:developfrom
daedalus:feature/implement-specs

Conversation

@daedalus

@daedalus daedalus commented May 8, 2026

Copy link
Copy Markdown

Summary

Changes

bloom-filter-v3

  • New mempalace/content_hash.pyBloomFilter + ContentHashDB classes
  • Modified mempalace/miner.py — integrated ContentHashDB into process_file() and mine()
  • Modified mempalace/convo_miner.py — integrated ContentHashDB into mine_convos()
  • New tests/test_hashdb.py — 12 tests (all passing)

git-miner

  • New mempalace/git_miner.py — git mining module (334 lines)
  • Modified mempalace/cli.py — added git mode, --branch argument
  • New tests/test_git_miner.py — 17 tests (all passing)

Usage

# Content hash dedup (automatic in mine/convos modes)
mempalace mine /path/to/project
mempalace mine /path/to/convos --mode convos

# Git miner
mempalace mine /path/to/repo --mode git
mempalace mine /path/to/repo --mode git --branch main --branch feature

Test Results

  • 29 new tests passing
  • All imports verified
  • Lint checks passed

- Add ContentHashDB with Bloom filter for content-aware deduplication
- Add git_miner module for mining git repos commit-by-commit
- Integrate ContentHashDB into miner.py and convo_miner.py
- Add --mode git and --branch CLI arguments
- Add tests for ContentHashDB (12 tests) and git_miner (17 tests)
@daedalus daedalus changed the title feat: implement bloom-filter-v3 and git-miner from SPECs feat: implement bloom-filter-v3 and git-miner May 8, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a git mining mode and content-based duplicate detection using a Bloom filter and SQLite. Feedback identifies memory concerns when hashing large files, logic errors in the file-skipping mechanism within miner.py and convo_miner.py, and inefficient storage of the Bloom filter. Additionally, it is recommended to chunk large git commit content to ensure better search performance and quality.

Comment thread mempalace/content_hash.py
Comment thread mempalace/miner.py Outdated
Comment thread mempalace/convo_miner.py Outdated
Comment thread mempalace/content_hash.py Outdated
Comment thread mempalace/git_miner.py Outdated
daedalus and others added 4 commits May 8, 2026 10:49
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- Save bloom filter as gzip-compressed pickle with SHA256 integrity hash
- Add SafeUnpickler that only allows basic built-in types
- Reject malicious pickle files that try to execute code
- Add tests for malicious pickle rejection and hash verification
@daedalus
daedalus marked this pull request as draft May 8, 2026 14:06
- Add chunk_diff() function to split large diffs into smaller pieces
- Process_commit now creates multiple drawers per commit (one per chunk)
- Add chunking constants: CHUNK_SIZE=2000, CHUNK_OVERLAP=200, MIN_CHUNK_SIZE=50
- Add tests for chunking functionality (TestChunkDiff: 4 tests)
- Update dry-run output to show chunk count
- Fix return value to reflect total drawers created
@daedalus
daedalus marked this pull request as ready for review May 8, 2026 14:11
@igorls igorls added enhancement New feature or request area/mining File and conversation mining labels May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/mining File and conversation mining enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants