feat(archive): prune observer frames past the retention window - #6507
Open
wpfleger96 wants to merge 3 commits into
Open
feat(archive): prune observer frames past the retention window#6507wpfleger96 wants to merge 3 commits into
wpfleger96 wants to merge 3 commits into
Conversation
Add the opportunistic prune worker for the local archive DB: it deletes kind-24200 observer-frame scope rows whose archived_at is older than observer_retention_days, cascading orphan GC to both derived indexes, and keeps every other kind forever. Triggers are startup + post-commit, detached from the archive write and collapsed by an in-process single-flight flag plus a >=24h archive_meta.prune_last_success_at gate into at most one scan per day. Candidate scope keys are materialized and fixed per batch transaction; the candidate-scoped GC cascades observer_channel_index (which the whole-scan GC does not cover) and agent_metric_index. The observer-index insert becomes a conditional INSERT ... SELECT so a stale backfill callback cannot resurrect an index row for a pruned event. The prune-candidate SQL constant is colocated with the worker so the EXPLAIN-shape test pins the shipped query. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
force-pushed
the
wpfleger/archive-retention-prune
branch
from
August 21, 2026 19:22
a76edfb to
15e0fdf
Compare
The prune worker's startup trigger hung off start_archive_sync, but the renderer skips start_archive_sync while observer reconciliation is not ready (useArchiveSync returns early), so an archive whose reconciliation never succeeds and takes no new commits was never pruned — violating the plan's startup + post-commit trigger contract. Move the startup trigger into spawn_warm_init, which runs on every application startup after the init barrier warms, and drop the now-redundant sync-start hook. Also make the partition worklist deterministic (ORDER BY identity/relay) so a mid-worklist failure leaves a well-defined pruned prefix, and rework the failure regression to fail only the later partition, assert the earlier committed with the gate unstamped, then clear the fault and prove the retry completes the remainder and stamps success. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
bool::then_some evaluates its argument eagerly, so every caller that lost the compare_exchange still constructed a PruneGuard and dropped it with the returned None. That Drop cleared prune_running under the winning pass, so a burst of triggers admitted multiple concurrent prunes (Gurney's live lane: 8 triggers -> 4 admitted, 3 escaped with SQLITE_BUSY). Construct the guard lazily with then(|| ..) so only the winning caller ever builds one. Add two regressions in archive_db_tests.rs: a synchronous exactly-once test where a second losing call proves the flag stays set under a held guard, and an async burst test mirroring the maybe_prune control flow proving exactly one pass enters. Reverting to then_some fails both. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2' (P2') of the local-archive retention arc — the opportunistic prune worker that bounds the archive DB, built on the merged P1' schema and gated adapter (#5719).
What this does
The worker deletes kind-24200 observer-frame scope rows whose
archived_atis older thanobserver_retention_days(fromarchive_meta), cascading orphan GC to both derived indexes. Every other archived kind — NIP-AM metrics (44200) and any custom subscription kind — is kept indefinitely and never touched, per Will's v4 ruling.Scheduling
No cron. The worker is triggered opportunistically — once shortly after application startup (from
spawn_warm_init, after the archive init barrier warms) and again after each archive commit — and gated two ways so those frequent triggers collapse into at most one real prune per day:compare_exchangeon anAtomicBool(owned byArchiveDb, alongside the maintenance lock) admits exactly one prune at a time; a trigger arriving mid-prune is dropped, not queued.archive_meta.prune_last_success_at— a prune that finds the last success within 24h returns immediately. The timestamp is stamped after every clean pass, including one that deleted nothing.The gate is database-global, so a due pass does global work: it enumerates a fixed worklist of every archived
(identity_pubkey, relay_url)partition and prunes each under the same cutoff, stamping success only after the whole worklist completes. It never scopes to the identity/relay that happened to be active when the trigger fired, so an inactive historical partition is not starved, and a mid-worklist failure leaves the gate unstamped for the idempotent pass to retry.The startup trigger runs on every application launch, independent of whether archive sync ever starts — the renderer skips
start_archive_syncwhile observer reconciliation is not ready, so hooking the startup pass there would miss an archive whose reconciliation never succeeds and which takes no new commits. A prune failure is fully detached from the archive commit that triggered it:trigger_prunespawns onto the async runtime and swallows the result, so a prune error can never fail a user's archive write.Delete discipline
Candidate scope-row primary keys are materialized inside each batch transaction and fixed for that transaction (binding condition #4): the batch deletes exactly those rows, then runs a candidate-scoped orphan GC over only the event ids it just unscoped — never a widened whole-table anti-join (
store::gc_orphaned_eventsstays a rare repair pass) and never a post-delete re-scan. The GC cascades BOTH derived indexes:observer_channel_index(which the whole-scan GC does not cover) andagent_metric_index. Deletes run in bounded batches so each transaction's write lock is brief.upsert_observer_channel_indexbecomes a conditionalINSERT ... SELECTfromarchived_eventsso a stale backfill callback cannot resurrect an index row for an event the prune already removed.The prune-candidate SQL constant is colocated with the worker (
PRUNE_CANDIDATE_SQLinprune.rs, previously only inretention_tests.rs) so the EXPLAIN-shape test pins the shipped query and the two cannot drift.Tests
prune_tests.rscovers cutoff correctness (incl. late-arriving-old and future-dated events on thearchived_atbasis), non-observer kinds kept forever, multi-scope events surviving on a within-window scope, dual-index cascade, multi-batch clearing, forced-error mid-batch rolling back fully with no orphaned index rows, the 24h gate (skip/run/never-pruned/empty-but-clean-stamps), a global pass pruning every stored partition before stamping success, a mid-worklist partition failure committing the earlier partition, leaving the gate unstamped, then a retry completing the remainder and stamping success, the stale-backfill conditional insert, the first-archiveON CONFLICT DO NOTHINGpin, and the EXPLAIN-shape assertion against the covering index (archived_at<?seek, no temp b-tree). Writer/pruner contention over the async single-flight wiring is exercised by the live lane.