feat: add mine_epoch tracking and revision snapshots

[YulelogPagoda]

What does this PR do?

Adds two features that build on top of #521's delete-before-insert fix, giving every mine run a version identity and preserving a queryable history of previous file versions.

1. Mine epoch tracking

Each mine() run increments a monotonic counter stored in {palace}/epoch.json. Every chunk filed during that run gets tagged with its mine_epoch in metadata. This gives every drawer a "generation number" that enables:

• Knowing how stale a chunk is (current_epoch - chunk.mine_epoch)
• Auditing which mine run introduced any given piece of content
• Reconstructing the palace state at any previous epoch

The status command now displays the current epoch.

2. Revision snapshots

When a file is re-mined, #521's delete-before-insert purges all existing chunks for that file before the fresh ones are written. This avoids hnswlib segfaults but also destroys the previous version of the file. This PR adds a snapshot step that runs just before the delete, capturing the chunks to {palace}/revisions.jsonl so the previous version can still be retrieved.

Each revision record captures:

• superseded_at / superseded_by_epoch — when and by which run
• source_file / chunk_index — what it was
• content — exact chunk text
• original_epoch / original_filed_at — provenance

Why this is valuable

The two features compose naturally:

• Epoch tracking tells you when something changed
• Revision snapshots tell you what it used to be
• Together they turn the palace into a versioned store without touching the active search index

How to test

11 tests, ~24s, no API keys, no network:

• 4 epoch tracking tests (starts at 0, increments, persists to JSON, tags chunks)
• 3 eviction behavior tests (shrinking, unchanged, growing)
• 3 revision snapshot tests (file created, content preserved, accumulates)
• 1 backward compat test (old signature still works)

Design notes

• No new dependencies — uses stdlib json + existing ChromaDB
• Backward compatible — process_file() accepts new args with defaults, existing drawers without mine_epoch metadata default to 0
• Non-intrusive — does NOT modify the existing delete-before-insert logic from hnswlib updatePoint race on modified-file re-mine: EXC_BAD_ACCESS in repairConnectionsForUpdate (macOS ARM64, Python 3.13, chromadb 0.6.3) #521, only adds around it
• Best-effort cleanup — snapshot wrapped in try/except so failures never break mining

Checklist

• Tests pass (python -m pytest tests/test_mine_epoch_tracking.py -v)
• No hardcoded paths
• Linter passes (ruff check .) — not run locally, please verify

[web3guru888]

This is a well-scoped feature pair — epoch tracking and revision snapshots compose cleanly and the implementation is non-intrusive.

What works well

• Epoch counter is monotonic and persisted atomically to epoch.json — simple, no external deps, survives crashes between runs.
• _snapshot_revisions() is wrapped in try/except so a write failure never breaks mining. Right call.
• The mine_epoch: int = 0 default on process_file() and add_drawer() means callers that don't pass palace_path get graceful degradation (epoch 0 in metadata, no snapshot attempt). The backward-compat test validates this explicitly.
• Test coverage is solid: epoch lifecycle, eviction behaviour, snapshot content integrity, accumulation across runs. The UNIQUE_MARKER_STRING_XYZZY_12345 round-trip test is particularly good — it proves the content is right, not just that a file was created.

One real concern — revisions.jsonl grows unboundedly

Every re-mine of a changing file appends records. A project that re-mines a large codebase every day will accumulate tens of thousands of lines quickly. There's no rotation, truncation, or max-size guard. A 50K-drawer palace that re-mines 10% of its files daily would write ~5K revision records per run — ~1.5 million records after a year, uncompressed.

Worth at least a MAX_REVISIONS constant (default ~50,000 lines) with a tail-truncation on open, or a note in the docstring that the file is intentionally unbounded and callers should manage it themselves. The --purge command from #562 is a natural place to reset revisions too.

Minor: epoch.json is inside the palace directory

This is fine for the common case, but palace directories may be version-controlled or synced. epoch.json and revisions.jsonl should probably be documented as ephemeral/local files that shouldn't be committed. A note in the PR description or a .gitignore suggestion for {palace}/*.jsonl and {palace}/epoch.json would help.

Interaction with #562's cmd_purge

The purge-and-rebuild command in #562 deletes and recreates the palace directory. After a purge, _load_epoch() returns 0 and the counter restarts from 1, which means epoch numbers become non-monotonic across purge boundaries. If continuity matters (audit trail), cmd_purge would need to preserve epoch.json (or at least record the last epoch before purge in the new one). Minor edge case, but worth a comment in cmd_purge.

Verdict: LGTM with the revisions.jsonl growth concern as the main thing to address before merge.

[igorls]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

[YulelogPagoda]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

I'll work up to 3.8.0 and roll back in 😊 I'm finding it a lovely useful thing to have. Thank you 😊