Skip to content
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
4098c54
feat(convo_miner): auto-route AI tool sessions to wing_api
milla-jovovich Apr 27, 2026
cb13036
fix(migrate): close SQLite connection and clean temp palace on exception
arnoldwender Apr 26, 2026
2c75c6f
Fix mine palace lock holder encoding
fatkobra May 10, 2026
c49c10f
Add regression tests for UTF-8 lock holder writes
fatkobra May 10, 2026
92de001
fix(convo_miner): scope skip-check and drawer ids by extract_mode (#1…
therahul-yo May 16, 2026
2bf3621
refactor(convo_miner): reuse _metadata_matches_extract_mode helper
therahul-yo May 16, 2026
419a34c
style(tests): restore ruff 0.4.x assert formatting in test_palace_loc…
igorls May 17, 2026
18bda00
Merge pull request #1438 from fatkobra/patch-1
igorls May 18, 2026
83e4711
style(tests): apply ruff 0.4.x format to test_convo_miner (#1528)
igorls May 18, 2026
5c9a5e0
Merge pull request #1528 from therahul-yo/fix/1505-extract-mode-aware…
igorls May 18, 2026
96a3272
ci(lint): pin ruff to 0.15.9 across CI and dev deps
igorls May 18, 2026
2cab6f5
style: reformat tree with ruff 0.15.9
igorls May 18, 2026
283ef03
ci(lint): pin ruff in pre-commit + uv.lock too (#1541 review)
igorls May 18, 2026
eee5f32
Write placeholder PID when claiming mine slot
fatkobra May 18, 2026
b201255
Refactor mine slot placeholder creation
fatkobra May 18, 2026
8f40dce
Merge pull request #1541 from MemPalace/ci/pin-ruff-0.15
igorls May 18, 2026
3cac26f
fix(convo_miner): enforce CHUNK_SIZE in paragraph chunker (#1534)
mvalentsev May 17, 2026
e811b6d
fix: avoid false HNSW metadata quarantine
dergachoff May 11, 2026
3da27d3
fix: VACUUM + FTS5 rebuild after repair to reclaim SQLite space
williamblair333 May 15, 2026
e328bf4
Merge pull request #1452 from dergachoff/fix/hnsw-missing-dim-quarantine
igorls May 18, 2026
d74ca06
Merge pull request #1523 from williamblair333/fix/repair-vacuum-fts5
igorls May 18, 2026
34d8dde
Merge pull request #1216 from arnoldwender/fix/migrate-resource-cleanup
igorls May 18, 2026
3e92719
fix(mcp): surface create_tunnel ValueError instead of masking it (#1473)
igorls May 18, 2026
426d164
test(hooks): isolate test_hooks_cli from test_cli side effect (#1510)
igorls May 18, 2026
02c05aa
refactor(mcp): merge the two ValueError guards in tool_create_tunnel
igorls May 18, 2026
90fece7
test(hooks): also isolate _MINE_PID_DIR + create the state dir (#1510)
igorls May 18, 2026
446e017
Merge pull request #1547 from MemPalace/fix/1510-hooks-cli-test-isola…
igorls May 18, 2026
2c9dd39
Merge pull request #1546 from MemPalace/fix/1473-tunnel-error-surface
igorls May 18, 2026
100539d
Merge pull request #1543 from fatkobra/fix/1443-mine-slot-placeholder
igorls May 19, 2026
1b94f4e
Merge pull request #1538 from mvalentsev/fix/1534-paragraph-chunk-siz…
igorls May 19, 2026
5decc0c
fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455)
mvalentsev May 19, 2026
22cec56
fix(miner): use Optional[int] for Python 3.9 compat (#1455)
mvalentsev May 19, 2026
81f3e32
fix(miner): consolidate skip accounting under --dry-run (#1455)
mvalentsev May 19, 2026
8d8a8c9
fix(miner): polish per multi-agent review (#1455)
mvalentsev May 19, 2026
5e3d3dd
feat(3.3.6): virtual line numbering + format coverage via --mode extract
milla-jovovich May 20, 2026
e362dfa
feat(format_miner): address bot review — parity fixes with miner.py
milla-jovovich May 20, 2026
5c40cc3
feat(format_miner): detect_room + topic tunnel computation (miner.py …
milla-jovovich May 20, 2026
0b9c562
fix(miner): mirror init's case-insensitive entity matching in per-dra…
milla-jovovich May 20, 2026
07807ab
feat(hallways): within-wing entity-to-entity connector primitive
milla-jovovich May 20, 2026
fc13be5
feat(miner): integrate compute_hallways_for_wing into post-mine flow
milla-jovovich May 20, 2026
923f797
Merge pull request #1560 from MemPalace/feat/integrate-hallways-in-miner
igorls May 20, 2026
a0ab873
Merge pull request #1557 from MemPalace/fix/miner-case-insensitive-en…
igorls May 20, 2026
2e29014
Merge pull request #1562 from MemPalace/igorls/fix-subprocess-hook-ch…
igorls May 20, 2026
7bddb65
fix: stale-PID timeout, MCP idle exit, structured errors (#1552)
igorls May 20, 2026
1c818d5
fix: address stale hook review feedback
igorls May 20, 2026
5f0a214
style: format hook tests
igorls May 20, 2026
498b22f
Merge pull request #1554 from mvalentsev/fix/1455-max-chunks-configur…
igorls May 20, 2026
c18879b
feat(tunnels): cross-wing entity tunnels derived from hallways
milla-jovovich May 20, 2026
ffc2ae4
fix(extract): surface MissingDependencyException + pull format sub-ex…
milla-jovovich May 20, 2026
ad875a8
Merge pull request #1558 from MemPalace/feat/hallways-within-wing-con…
igorls May 21, 2026
ed80f80
Merge pull request #1565 from MemPalace/feat/entity-tunnels-from-hall…
igorls May 21, 2026
95caf80
Merge pull request #1555 from MemPalace/feat/3.3.6-line-numbers-and-f…
igorls May 21, 2026
b6dc122
fix(extract): polish PR — address bot review feedback on PR #1555
milla-jovovich May 21, 2026
60d460b
Merge pull request #1236 from MemPalace/feat/convo-miner-wing-api-aut…
milla-jovovich May 21, 2026
be64371
Merge pull request #1572 from MemPalace/fix/pr-1555-polish-bot-feedback
milla-jovovich May 21, 2026
6658a4d
fix(audit): chunk and batch content before embedding upsert (#1539)
mvalentsev May 17, 2026
fa7f87f
Merge pull request #1540 from mvalentsev/fix/1539-three-unbounded-ups…
igorls May 21, 2026
b1f7bab
merge: upstream/develop into fork main (57 commits, last sync 2026-05…
jphein May 21, 2026
e8cdecd
style: collapse implicit string concat in palace_graph warning (ruff …
jphein May 22, 2026
827de69
fix(merge): include warnings in single-chunk tool_add_drawer return
jphein May 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,10 @@ jobs:
with:
python-version: "3.11"
cache: 'pip'
- run: pip install "ruff>=0.4.0,<0.5"
# Keep this pin identical to the `ruff==` pin in pyproject.toml
# ([project.optional-dependencies].dev and [dependency-groups].dev)
# so CI and `pip install -e ".[dev]"` format/lint identically.
- run: pip install "ruff==0.15.9"
- run: ruff check .
- run: ruff format --check .

Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,5 @@ scratch/
# Local benchmark artifacts (large jsonl dumps, per-question stdout)
benchmarks/c_beta/*.jsonl
benchmarks/c_beta/*.stdout

.review-logs
9 changes: 5 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Keep in lock-step with the ruff version pinned in .github/workflows/ci.yml
# (>=0.4.0,<0.5). Using a newer rev here produces a different formatter
# output than CI and breaks `ruff format --check` in the lint job.
rev: v0.4.10
# Keep this rev in lock-step with the exact ruff pin in pyproject.toml
# ([project.optional-dependencies].dev / [dependency-groups].dev) and
# .github/workflows/ci.yml. A different ruff here produces different
# formatter output than CI and breaks `ruff format --check` in lint.
rev: v0.15.9
hooks:
- id: ruff
args: [--fix]
Expand Down
19 changes: 8 additions & 11 deletions benchmarks/mine_bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,7 @@ def build_corpus(dest: Path, n_files: int, paragraphs_per_file: int, seed: int)
paragraphs.append(" ".join(words))
(dest / f"doc_{i:03d}.md").write_text("\n\n".join(paragraphs))
(dest / "mempalace.yaml").write_text(
"wing: bench\n"
"rooms:\n"
" - name: general\n"
" description: all\n"
" keywords: [general]\n"
"wing: bench\nrooms:\n - name: general\n description: all\n keywords: [general]\n"
)


Expand Down Expand Up @@ -121,8 +117,7 @@ def _process_file_unbatched(filepath, project_path, collection, wing, rooms, age
]
closet_lines = build_closet_lines(source_file, drawer_ids, content, wing, room)
closet_id_base = (
f"closet_{wing}_{room}_"
f"{hashlib.sha256(source_file.encode()).hexdigest()[:24]}"
f"closet_{wing}_{room}_{hashlib.sha256(source_file.encode()).hexdigest()[:24]}"
)
closet_meta = {
"wing": wing,
Expand Down Expand Up @@ -155,7 +150,7 @@ def mine_once(project_dir: str, palace_path: str, batched: bool) -> tuple[int, f
t0 = time.perf_counter()
for filepath in files:
if batched:
drawers, _ = miner.process_file(
drawers, _, _ = miner.process_file(
filepath=filepath,
project_path=project_path,
collection=collection,
Expand Down Expand Up @@ -217,9 +212,9 @@ def run_scenario(label: str, n_files: int, paragraphs_per_file: int, seed: int)


SCENARIOS = {
"small": ("Small files (~50 paragraphs)", 10, 50),
"small": ("Small files (~50 paragraphs)", 10, 50),
"medium": ("Medium files (~200 paragraphs)", 20, 200),
"large": ("Large files (~500 paragraphs)", 10, 500),
"large": ("Large files (~500 paragraphs)", 10, 500),
}


Expand All @@ -237,7 +232,9 @@ def _env_summary(device_label: str) -> list[str]:
import onnxruntime as ort

ort_v = ort.__version__
providers = ",".join(p.replace("ExecutionProvider", "") for p in ort.get_available_providers())
providers = ",".join(
p.replace("ExecutionProvider", "") for p in ort.get_available_providers()
)
except Exception:
ort_v = "?"
providers = "?"
Expand Down
223 changes: 223 additions & 0 deletions docs/format-coverage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
# Format coverage — `mempalace mine --mode extract`

**Shipped in mempalace 3.3.6 (PR #1555).**
**Module:** `mempalace/format_miner.py`
**CLI entry:** `mempalace/cli.py` — `cmd_mine` dispatcher routes `--mode extract` here.
**Tests:** `tests/test_format_miner.py` — covers all 14 fringe cases plus orchestrator behavior. See the file for the current test inventory (the count drifts as polish PRs land).

---

## What it does

Adds a third miner alongside the existing two:

```
mempalace mine <dir> → miner.py (project files: code, docs, notes)
mempalace mine <dir> --mode convos → convo_miner.py (chat exports)
mempalace mine <dir> --mode extract → format_miner.py (binary office documents) ← NEW
```

Supported formats and the transformer each routes through:

| Extension | Transformer | Python | License |
|---|---|---|---|
| `.pdf` | MarkItDown | ≥ 3.10 | MIT |
| `.docx` | MarkItDown | ≥ 3.10 | MIT |
| `.pptx` | MarkItDown | ≥ 3.10 | MIT |
| `.xlsx` | MarkItDown | ≥ 3.10 | MIT |
| `.epub` | MarkItDown | ≥ 3.10 | MIT |
| `.rtf` | **striprtf** | ≥ 3.6 | MIT |

**Per-format routing — why `.rtf` doesn't go through MarkItDown.** Verified live on 2026-05-19 against a local mixed-format test directory: MarkItDown 0.1.5 does NOT convert `.rtf`. It returns the raw RTF control-code source unchanged (`{\rtf1\ansi\ansicpg1252...`). For RTF input, format_miner routes to [striprtf](https://github.com/joshy/striprtf) (pure-Python, MIT, ~150 lines, purpose-built for RTF → plain text). The principle stays intact: transform at read time, never modify the source — only the choice of transformer is per-format.

Both libraries are **optional** runtime dependencies. Users who only mine projects or convos don't pay either install footprint. Users on Python 3.9 can still install `striprtf` to extract RTF; MarkItDown requires 3.10+ and silently skips if not present. Per-extension install messages route to the right library:

- Missing MarkItDown (any of pdf/docx/pptx/xlsx/epub) → `SKIP_NO_MARKITDOWN` → log: `pip install markitdown`
- Missing striprtf (any `.rtf`) → `SKIP_NO_STRIPRTF` → log: `pip install striprtf`

---

## Design — read-time conversion, never modifies source files

Same architectural principle as virtual line numbering (the other 3.3.6 feature in this package): **transform at read time, leave storage untouched.**

```
PDF on disk → (read bytes into memory) → MarkItDown → Markdown text → existing chunker → drawers in ChromaDB
```

- The PDF on the user's disk is read but **never modified**.
- No intermediate `.md` files are written to the user's filesystem.
- The conversion lives entirely in memory; once the text reaches the existing chunker, it's handled identically to any other ingestion.
- Closets / drawers / halls all work the same — the format miner produces the same drawer-shape as the other miners.

---

## Why a new miner, not edits to `miner.py`

Three reasons, in order of importance:

1. **Matches the existing miner-per-content-type pattern.** Mempalace already has `miner.py` and `convo_miner.py`. A third miner is a third instance of the same shape — immediately recognizable to anyone reading the codebase.
2. **Smaller review surface.** `format_miner.py` is a pure addition. Reviewer reads one new file in isolation; the stable code paths in `miner.py` and `convo_miner.py` are untouched.
3. **Cleaner failure isolation.** If the format miner has a bug, the existing miners are physically incapable of being affected.

The trade-off vs. integrating into `miner.py` was decided during the 2026-05-19 design pass: **the new-miner pattern wins on architecture and on review surface.**

---

## Fringe cases handled — 13 (original 12 + striprtf parity)

Each case maps to a specific `ExtractionStatus` enum value so callers (and reviewers) can audit exactly which path fired. All thirteen are covered by dedicated tests in `tests/test_format_miner.py`.

| # | Case | Status code | Behavior |
|---|---|---|---|
| 1 | MarkItDown not installed (any non-RTF format) | `SKIP_NO_MARKITDOWN` | Logs `pip install markitdown` instruction; skips file |
| 2 | File too large (> 500 MB default; caller can override) | `SKIP_TOO_LARGE` | Logs size and threshold; skips |
| 3 | iCloud cloud-only file (`.icloud` suffix OR `st_flags` dataless bit) | `SKIP_CLOUD_ONLY` | Detected before any I/O that would trigger materialization; skips |
| 4 | Encrypted / password-protected PDF | `SKIP_ENCRYPTED` | Catches exceptions matching `encrypt`/`decrypt`/`password`/`protected`; skips |
| 5 | Empty file (zero bytes) | `SKIP_EMPTY` | Silent skip |
| 6 | Permission denied | `SKIP_PERMISSION` | Catches `PermissionError`; logs; skips |
| 7 | Broken symlink (target missing) | `SKIP_BROKEN_SYMLINK` | Detected via `is_symlink()` + `not exists()`; skips |
| 8 | Dirty encoding in extracted text | (recovered) | `decode_robust()` falls back UTF-8 → CP1252 → UTF-8-with-replace; never raises |
| 9 | Windows path semantics | (no special status) | `pathlib.Path` throughout; case-insensitive suffix matching; accepts `str` or `Path` input |
| 10 | Transformer internal crash on malformed file | `SKIP_EXTRACTION_ERROR` | Catches generic `Exception` at file boundary; logs filename + exception type + message tail; skips |
| 11 | Network / sync-drive timeout | `SKIP_NETWORK_TIMEOUT` | Catches `TimeoutError`; skips |
| 12 | Unrecognized extension | `SKIP_UNRECOGNIZED` | Cheap suffix-set check; skips |
| 13 | **striprtf not installed (any `.rtf`)** | `SKIP_NO_STRIPRTF` | Logs `pip install striprtf` instruction; skips file. Added 2026-05-19 after a live integration test surfaced MarkItDown 0.1.5's lack of RTF support. |

Plus `SKIP_UNREADABLE` for catch-all `OSError` during stat (not in the original list, added for defensive completeness — see file-stat block in `extract_text`).

---

## Cases NOT solved by this miner (deferred)

Per the spec, these are documented limits, not bugs:

- **Custom PDF parsers for specific document types.** MarkItDown does its best; we accept its limits.
- **OCR on scanned image PDFs.** Separate concern, opt-in feature for a later release.
- **DRM-locked files.** We can't bypass DRM and shouldn't try.
- **Pathological corrupt files.** They're corrupt — `SKIP_EXTRACTION_ERROR` with a clear log line is the responsible outcome.

These limitations are reported per-file via the skip-code log, so the user always knows what didn't make it in and why.

---

## API reference

All public symbols are exported via `__all__`. Two functions and one enum form the contract:

### `extract_text(path, max_file_size=DEFAULT_MAX_FILE_SIZE) -> tuple[Optional[str], ExtractionStatus]`

Convert one file to text. Returns `(text, status)` where `text` is the extracted Markdown for `OK` cases and `None` for every skip. Pure function — source file at `path` is never modified.

Accepts both `Path` and `str` for `path`.

### `scan_formats(directory) -> list[Path]`

Walk `directory` recursively, return supported files sorted by path. Skips hidden / build directories (`.git`, `.venv`, `__pycache__`, etc.) and OS metadata files (`.DS_Store`, `Thumbs.db`, `desktop.ini`).

### `ExtractionStatus` (Enum)

Twelve documented values plus `SKIP_UNREADABLE`. Each is paired with a string value like `"skip:no_markitdown"` for log readability; tests assert on `.name` for stability.

### `decode_robust(raw: bytes) -> str`

Identical to the helper in the terminal-session normalizer. UTF-8 first, CP1252 fallback, UTF-8-with-replace as final safety net. Never raises.

### `is_icloud_dataless(path: Path) -> bool`

Two signals:
1. Literal `.icloud` suffix (iCloud's offloaded-file placeholder convention)
2. macOS `st_flags` dataless bit (`0x40000000`) — best-effort, gracefully degrades on non-macOS

Returning `True` causes `extract_text` to short-circuit to `SKIP_CLOUD_ONLY` before any I/O that would trigger iCloud materialization.

---

## CLI integration

One-line addition to `mempalace/cli.py` in `cmd_mine`:

```python
def cmd_mine(args):
...
try:
if args.mode == "convos":
from .convo_miner import mine_convos
mine_convos(...)
elif args.mode == "extract": # ← NEW
from .format_miner import mine_formats # ← NEW
mine_formats( # ← NEW
format_dir=args.dir, # ← NEW
palace_path=palace_path, # ← NEW
wing=args.wing, # ← NEW
agent=args.agent, # ← NEW
limit=args.limit, # ← NEW
dry_run=args.dry_run, # ← NEW
) # ← NEW
else:
from .miner import mine
mine(...)
```

And add `"extract"` to the `--mode` choices in the argparse setup:

```python
p_mine.add_argument(
"--mode",
choices=["projects", "convos", "extract"], # ← extract added
default="projects",
help="Ingest mode: 'projects' (default), 'convos' for chat exports, 'extract' for office documents",
)
```

That's the entire CLI change. Two argparse edits + one `elif` branch.

---

## Empirical smoke test — local mixed-format directory

Run on a local test directory containing **52 `.rtf` files + 11 `.pdf` files** (a mix of long-form correspondence and research/architecture documents, no synthetic fixtures):

```
Supported formats: ['.docx', '.epub', '.pdf', '.pptx', '.rtf', '.xlsx']
scan_formats: found 63 supported files
```

**Live end-to-end extraction (2026-05-19, Python 3.13, MarkItDown 0.1.5 + striprtf 0.0.27):**

| Format | Files | Result | Chars extracted |
|---|---|---|---|
| RTF (via striprtf) | 52 | **52/52 OK** | 1,350,679 |
| PDF (via MarkItDown) | 11 | 10 OK, 1 SKIP_ENCRYPTED | 1,340,869 |

The single PDF skip is a real password-protected file — Fringe Case 4 (`SKIP_ENCRYPTED`) fired correctly on real data via the exception-message matcher. Designed behavior validated empirically.

Anti-regression assertion: **zero raw `\rtf1` / `\ansi` control codes leak into extracted text**. Without the per-format routing fix, 52 of 52 RTFs would have ingested as unreadable control-code source. With the fix, every RTF yields clean plain text. This is the integration bug that mocked unit tests cannot catch and that motivated the `SKIP_NO_STRIPRTF` status + per-format dispatch.

---

## Backwards compatibility

- **No changes to existing miners.** `miner.py` and `convo_miner.py` are not touched.
- **No new required dependencies.** MarkItDown and striprtf are optional extras (`pip install markitdown striprtf` or `pip install mempalace[extract]` if declared as such in `pyproject.toml`). MarkItDown requires Python ≥ 3.10; striprtf works on 3.6+. Recommended pyproject entries use environment markers so 3.9 users automatically get only striprtf:
```toml
[project.optional-dependencies]
extract = [
"striprtf>=0.0.27",
"markitdown>=0.1.5; python_version >= '3.10'",
]
```
- **No on-disk format changes.** Drawers / closets / halls have the exact same shape regardless of which miner produced them.
- **No migration step.** Users who already have a palace mined from the other modes can run `mempalace mine --mode extract ~/docs/` and the new drawers land in the same palace, same wing, same room conventions.
- **Optional `--max-file-size` flag** can be added to `mempalace mine` so users with legitimate large files (e.g. scanned books) can raise the cap from the 500 MB default.

---

## Out of scope for 3.3.6

- **Streaming extraction for huge PDFs** (>500 MB). MarkItDown's default behavior is whole-file in memory. For users with multi-GB PDFs, a streaming path via `pypdf` page-by-page is the natural extension. Deferred to 3.3.7 or 3.4.0.
- **Audio / video transcription.** Whisper integration is its own scope.
- **OCR for scanned PDFs.** Tesseract / cloud OCR integration is its own scope.
- **Migration importers** (Notion / Obsidian / Mem0 export formats). Each is a separate adapter; not bundled here.

These are real future features; they don't belong in a 3.3.6 release whose scope is "additive format coverage via MarkItDown."
Loading
Loading