fix(miner): close four re-mine safety gaps in process_file

[KeilerHirsch]

Problem

Four related gaps in miner.py::process_file's re-mine path, all found in the same audit pass:

1. No completion marker across multi-batch upserts. process_file deletes all existing drawers for source_file, then re-inserts new chunks in batches of DRAWER_UPSERT_BATCH_SIZE. Every chunk in a pass carries the same source_mtime. If the process is killed after batch 1 commits but before a later batch, the palace ends up with only the first N batches, and old closets left dangling. On the next mempalace mine run, file_already_mined(check_mtime=True) finds the surviving batch's drawers with a source_mtime that still matches the file's on-disk mtime (the file itself was never touched) and reports the file as fully mined — the missing chunks become permanently unreachable.

2. source_mtime captured after content is read, not at read time (TOCTOU). The file is read once (_read_text_no_follow), then source_mtime is captured later via a separate os.path.getmtime() call — after chunking, room detection, and acquiring the mine lock. If the file is modified in that window (the exact scenario the codebase's own docs describe for an actively-appended Claude Code session log), the stored source_mtime reflects content that was never actually chunked. The next mine sees the current on-disk mtime match the stored value and treats the file as current — the appended tail is silently, permanently skipped.

3. Failed drawer purge swallowed to a debug log. collection.delete(where={"source_file": source_file}) is wrapped in except Exception: logger.debug(...), and execution continues unconditionally into the upsert loop. A transient delete failure (lock contention, a corrupted segment) then leaves old and new drawers coexisting — orphaned tail entries if the new set is smaller, silently overwritten rows if it's larger — with no operator-visible signal.

4. Old-drawer delete runs before the new chunk count is known. The delete at the top of the locked section runs unconditionally; the closet purge+rebuild only runs when drawers_added > 0. If every chunk from a re-mine ends up filtered below min_chunk_size after boundary-splitting, the old drawers are gone but the old closets survive, now pointing at deleted drawer IDs.

Fix

• Every chunk's metadata now carries chunk_total (the number of chunks this pass expects to write). file_already_mined verifies a matching-stored_mtime group's drawer count reaches chunk_total before reporting True. A drawer with no chunk_total (legacy rows, or the single-shot add_drawer() helper, which has no partial-batch risk) is trusted as before — this stays backward compatible.
• _read_text_no_follow now returns (content, mtime) from the same fstat() call that validates the file. process_file threads that single value through for the stored source_mtime instead of a later re-stat.
• A failed purge now aborts the file's mine attempt (return 0, room, None) and prints a stderr warning, instead of proceeding. The old drawers' stored mtime is untouched, so the next mine still sees a mismatch and retries.
• purge_file_closets now runs whenever the delete-and-rebuild cycle does, regardless of the new chunk count. Only the closet rebuild itself stays conditional on drawers_added > 0.

Tests

Five new tests in tests/test_miner.py (one direct unit test of file_already_mined's new completeness check, four exercising process_file via a FakeCol test double), plus the existing _metadata/_build_drawer_metadata coverage. Each new test was confirmed to fail against the pre-fix code (verified via git stash of the source files only) and pass after the fix. Full existing suite across test_miner.py, test_convo_miner*.py, test_palace.py, test_hallways.py, test_format_miner.py, and test_miner_fts5_validation.py still passes — no regressions.

[fatkobra]

Blocking: the new completion marker covers all drawer batches, but not the
closet phase that follows them.

By the time purge_file_closets() and upsert_closet_lines() run, every
drawer already carries the current source_mtime and the complete
chunk_total. file_already_mined() will therefore return True on the
next run.

This leaves two partial-completion cases that are not retried:

1. purge_file_closets() still catches every exception and logs only at
   debug level. If the purge fails, stale higher-numbered closets can survive
   while the new closets are written.
2. If upsert_closet_lines() raises after the drawer batches complete, the
   next mine sees the drawers as complete and skips the source instead of
   retrying the closet rebuild.

Please make the completion contract cover the entire per-file replacement,
not only the drawer upserts. For example, record final completion only after
the closet purge/rebuild succeeds, or otherwise ensure that a closet-stage
failure causes the next freshness check to return False.

Regression coverage should separately simulate:

• a failed closet purge followed by another mining run;
• a failed closet upsert followed by another mining run;

and verify that the second run retries the source rather than treating it as
fully complete.

[mvalentsev]

Hey @KeilerHirsch, one more thing about the chunk_total marker, pointing the other way from @fatkobra's closet-phase review: it doesn't survive per-drawer deletion.

dedup_source_group removes individual drawers inside one source_file group, so the survivors keep chunk_total=16 while only 6 rows are left, and file_already_mined can't satisfy that group again. Mining one file into a fresh palace on this head gives 16 drawers stamped chunk_total=16; python -m mempalace.dedup --threshold 0.15 leaves 6; with the file untouched on disk the next mine files 16 back, the same ids dedup had removed, where develop skips the file. Dedup then removes them again, so the two keep undoing each other.

Dedup is the path I ran, but prune_corrupt and the stale sweep in mcp_server delete by drawer id too, so the marker breaks the same way there.