perf: batch writes, bulk mtime dedup, concurrent mining, entity STOPWORDS

[jphein]

Split from #562 per maintainer request.

Changes

palace.py

• Epsilon mtime comparison — abs(stored - current) < 0.01 instead of ==. JSON round-trips lose float precision; strict equality caused unnecessary re-mining.
• Cosine distance space — create_collection(..., metadata={"hnsw:space": "cosine"}) so new palaces get cosine similarity by default.
• bulk_check_mined() — Pre-fetch all source_file/source_mtime pairs in paginated batches. Eliminates the N+1 query pattern where each file triggered a separate ChromaDB lookup.

miner.py

• Batch ChromaDB upserts — One collection.upsert() per file instead of per chunk. Reduces write overhead ~10x on typical files.
• ThreadPoolExecutor concurrent mining — --workers flag (default: min(8, cpu_count)) parallelizes file read/chunk/route while keeping DB writes sequential (ChromaDB client is not thread-safe).
• _prepare_file() helper — Pure-computation half of process_file(), safe for concurrent use.
• bulk_check_mined() integration — Single bulk fetch replaces per-file dedup queries in the concurrent path.
• Junk file filter — SKIP_PATTERNS (minified JS, lockfiles, source maps) and JUNK_FILE_SIZE (500 KB) skip files that are technically text but useless for semantic search.
• Word-boundary keyword matching — detect_room() uses re.findall(r'\b...\b') instead of substring count(), preventing false room routing (e.g., "test" matching "contest").
• Configurable chunk params — chunk_text() accepts optional chunk_size, chunk_overlap, min_chunk_size overrides.
• Paginated status() — Reads metadata in batches instead of a single limit=10000 call, fixing the 10K-drawer ceiling.

convo_miner.py

• Batch upserts — Accumulates chunks per file, then upserts in groups of 100 instead of one collection.upsert() per chunk.

entity_detector.py

• 73 technical STOPWORDS — Handler, Node, Service, Manager, Client, etc. These appear capitalized in technical docs but are not entities, causing false positives in entity detection.

cli.py

• --workers argument on mine subcommand, passed through to mine().

Test plan

• python -m pytest tests/ -x -q — all existing tests pass
• mempalace mine <dir> — verify batch writes produce identical palace content
• mempalace mine <dir> --workers 1 — sequential path still works
• mempalace mine <dir> --workers 4 — concurrent path works, no segfaults
• mempalace status — verify pagination works past 10K drawers
• mempalace mine <dir> --dry-run — dry run still reports correctly

[Copilot]

Pull request overview

This PR focuses on improving mining throughput and reducing redundant ChromaDB work by batching reads/writes, adding a bulk “already mined” prefetch, and enabling concurrent file preparation during project mining.

Changes:

• Updated Chroma collection creation and file mtime dedup logic, including a new bulk_check_mined() to avoid per-file lookups.
• Reworked project mining to support batched upserts and optional multi-worker concurrent preparation (--workers).
• Batched conversation mining upserts and expanded entity detector STOPWORDS to reduce false positives.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
mempalace/palace.py	Adds cosine-space default on collection creation; introduces bulk prefetch of mined mtimes.
mempalace/miner.py	Adds skip filters, word-boundary routing, configurable chunking, concurrent mining, and paginated status.
mempalace/convo_miner.py	Switches from per-chunk upserts to batched upserts per file.
mempalace/entity_detector.py	Extends STOPWORDS list with common technical/documentation terms.
mempalace/cli.py	Adds --workers flag to the mine subcommand and passes it through.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

bulk_check_mined() assumes every entry in batch["metadatas"] is a dict, but Chroma can return None/empty metadata entries. A single None will raise AttributeError (meta.get) and abort the bulk fetch early, defeating the N+1 elimination. Consider skipping non-dict metadatas (e.g., if not isinstance(meta, dict): continue) and continuing the pagination loop rather than bailing out of the whole function.

Suggested change

	for meta in batch["metadatas"]:
	src = meta.get("source_file")
	mtime = meta.get("source_mtime")
	if src and mtime is not None:
	mined[src] = float(mtime)
	if not batch["ids"]:
	break
	offset += len(batch["ids"])
	for meta in batch.get("metadatas") or []:
	if not isinstance(meta, dict):
	continue
	src = meta.get("source_file")
	mtime = meta.get("source_mtime")
	if src and mtime is not None:
	mined[src] = float(mtime)
	batch_ids = batch.get("ids") or []
	if not batch_ids:
	break
	offset += len(batch_ids)

[Copilot]

mine() reads chunk sizing from MempalaceConfig (palace_config.chunk_size/chunk_overlap/min_chunk_size), but MempalaceConfig currently doesn’t define these properties. This will raise AttributeError at runtime the first time mine() is called. Either add these config properties (with defaults) to mempalace/config.py or fall back to the module constants (CHUNK_SIZE/CHUNK_OVERLAP/MIN_CHUNK_SIZE) when the config doesn’t provide overrides.

Suggested change

	cfg_chunk_size = palace_config.chunk_size
	cfg_chunk_overlap = palace_config.chunk_overlap
	cfg_min_chunk_size = palace_config.min_chunk_size
	cfg_chunk_size = getattr(palace_config, "chunk_size", CHUNK_SIZE)
	cfg_chunk_overlap = getattr(palace_config, "chunk_overlap", CHUNK_OVERLAP)
	cfg_min_chunk_size = getattr(palace_config, "min_chunk_size", MIN_CHUNK_SIZE)

[Copilot]

Concurrent mining path upserts prepared chunks without first deleting existing drawers for that source_file. This bypasses the delete+insert safety in process_file() (added to avoid hnswlib updatePoint segfaults and to prevent stale drawers when a file’s chunk count shrinks). Consider issuing collection.delete(where={"source_file": str(filepath)}) for each processed file before its docs are added/flushed, or restructuring writes to preserve the same per-file delete+insert behavior in workers>1 mode.

[Copilot]

New workers>1 concurrent mining behavior (bulk_check_mined filtering + ThreadPoolExecutor prepare + batched upserts) isn’t covered by tests. Since tests/test_miner.py already exists for mine()/scan_project(), it would be valuable to add at least one integration test that runs mine(..., workers=2) against a temp project and asserts drawers are written and modified-file re-mines don’t leave stale drawers.

[Copilot]

_ADD_BATCH_SIZE is redefined on every file iteration. Since it’s a constant, consider moving it to module scope (or at least defining it once outside the per-file loop) to keep the hot path simpler and avoid repeated rebinding.