feat(store): sweep to reclaim orphaned legacy shared blobs

[getappz]

Follow-up to #351 (item #385).

Why

#351 gave each database its own blob directory and made deletes never touch the legacy shared ~/.agentflare/blobs — reads still fall back to it, and only a check against every database in the directory can prove a file there is dead. That was the right trade (leaking a file is recoverable; deleting one another store needs is not), but it means nothing reclaims that directory in the normal course of things.

On this install the dry run finds 107 of 372 legacy files unreferenced (538 KB).

What

agentflare_store::maintenance::sweep_legacy_blobs(root, dry_run) collects the referenced hashes from every .db in root, then removes the files in root/blobs/ that none of them names.

Safety properties, each with a test:

• Refuses to run with no database to consult. An empty reference set would make every file look dead — exactly when deleting is unrecoverable.
• Aborts rather than deletes if a database cannot be read. An incomplete reference set is worse than no sweep.
• A .db without a store_blobs table is skipped as unrelated, checked via sqlite_master rather than by swallowing a failed query — so a database that is broken rather than unrelated still surfaces instead of silently contributing nothing.
• Databases are opened SQLITE_OPEN_READ_ONLY and without migrations: the sweep must not alter what it only consults.

CLI

agentflare docs sweep-legacy-blobs [--dry-run]

Lives under docs because that is where the cache commands are and the docs database is what wrote most of these files, but the sweep consults store.db too, so it is safe regardless of which store owns a given blob. Output is the JSON report.

Real run against this install:

{ "scanned": 372, "reclaimed": 107, "bytes_reclaimed": 551024,
  "databases": ["flare-docs.db", "store.db"], "dry_run": true }

Verification

cargo fmt --all --check, the CI clippy invocation (--locked --workspace --all-targets --all-features -- -D warnings -A unsafe_code -A clippy::pedantic), and cargo test --workspace are all clean. Four new tests in maintenance.rs; the CLI path was exercised end to end with --dry-run against the real ~/.agentflare.

One small addition outside the sweep: agentflare_store::Error gained an Io(#[from] std::io::Error) variant, which the directory walk needs.

Summary by CodeRabbit

• New Features

  • Added a CLI command to scan legacy shared blob storage and reclaim orphaned files.
  • Added a --dry-run option to preview which files would be reclaimed without deleting them.
  • Added JSON output reporting scanned files, reclaimed files/bytes, and referenced databases.

• Bug Fixes

  • Improved storage maintenance error handling by supporting transparent I/O error reporting.
  • Ensures non-store SQLite files are ignored and no deletions occur when no store databases are found.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c4b66341-5119-4116-968f-040e2496c8ae

📥 Commits

Reviewing files that changed from the base of the PR and between c1f120a and 677fea8.

📒 Files selected for processing (2)

• crates/agentflare-store/src/maintenance.rs
• src/cli/docs.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• src/cli/docs.rs
• crates/agentflare-store/src/maintenance.rs

---

📝 Walkthrough

Walkthrough

Adds database-aware sweeping of legacy shared blobs, dry-run reporting, reclamation tests, IO error conversion, and a SweepLegacyBlobs CLI subcommand that emits JSON results.

Changes

Legacy blob sweep

Layer / File(s)	Summary
Sweep engine and reference tracking crates/agentflare-store/src/lib.rs, crates/agentflare-store/src/maintenance.rs	Adds IO error conversion, sweep reporting, referenced-hash collection across store databases, orphan deletion, dry-run handling, and tests for database and filesystem cases.
Sweep CLI integration src/cli/docs.rs	Adds the SweepLegacyBlobs command with --dry-run, invokes the sweep using the derived legacy root, and prints its JSON report.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DocsCLI
  participant sweep_legacy_blobs
  participant StoreDatabases
  participant LegacyBlobDirectory
  DocsCLI->>sweep_legacy_blobs: invoke with root and dry_run
  sweep_legacy_blobs->>StoreDatabases: collect referenced blob hashes
  sweep_legacy_blobs->>LegacyBlobDirectory: scan shard files
  sweep_legacy_blobs->>LegacyBlobDirectory: delete unreferenced files when not dry-run
  sweep_legacy_blobs-->>DocsCLI: return JSON LegacySweepReport

Loading

Possibly related PRs

• getappz/agentflare#244: Introduced the agentflare-store error enum extended by this change.
• getappz/agentflare#316: Also extends src/cli/docs.rs command parsing and dispatch.
• getappz/agentflare#351: Introduced blob-directory handling related to cleanup of the legacy shared blob directory.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: reclaiming orphaned legacy shared blobs in the store.
Description check	✅ Passed	The description covers purpose, behavior, CLI usage, safety properties, and verification, though its headings differ from the template.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/385-legacy-blob-sweep

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/agentflare-store/src/maintenance.rs`:
- Around line 324-338: Preserve the sweep safety boundary across all affected
sites: in crates/agentflare-store/src/maintenance.rs:324-338, propagate
individual read_dir entry errors and require at least one successfully consulted
actual store_blobs database rather than any .db file; in src/cli/docs.rs:96-102,
dispatch SweepLegacyBlobs before opening DocsStore and lazily open the store
only for commands that require it; in
crates/agentflare-store/src/maintenance.rs:491-513, add coverage confirming a
legacy blob directory containing only unrelated databases is refused without
deleting blobs.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0be924de-5adf-433e-8066-65efd8b80a35

📥 Commits

Reviewing files that changed from the base of the PR and between 527a4c8 and c1f120a.

📒 Files selected for processing (3)

• crates/agentflare-store/src/lib.rs
• crates/agentflare-store/src/maintenance.rs
• src/cli/docs.rs

[getappz]

Review pass — CodeRabbit's critical finding was valid on all three sites. Fixed in 677fea8 before merge.

The dbs.is_empty() refusal was unreachable from the only caller: docs::run opened DocsStore before dispatching, which creates and migrates a database in the very directory the sweep treats as its evidence. A home with a legacy blobs/ but no store left would gain a fresh empty one, yield an empty reference set, and delete every legacy blob — unrecoverably.

Consult first, then refuse — and count stores, not .db files

Checking the raw .db count also passed a directory holding only unrelated databases, whose reference set is just as empty.

-    let mut dbs: Vec<PathBuf> = std::fs::read_dir(root)?
-        .filter_map(|e| e.ok().map(|e| e.path()))
-        .filter(|p| p.extension().is_some_and(|ext| ext == "db"))
-        .collect();
+    // An entry that fails to read is propagated rather than skipped: it could
+    // be a store database, and silently dropping one from the reference set is
+    // what turns another store's live content into apparent garbage.
+    let mut dbs: Vec<PathBuf> = Vec::new();
+    for entry in std::fs::read_dir(root)? {
+        let path = entry?.path();
+        if path.extension().is_some_and(|ext| ext == "db") {
+            dbs.push(path);
+        }
+    }
     dbs.sort();
 
-    // With no database to consult, every file would look unreferenced.
-    if dbs.is_empty() {
+    let referenced = collect_referenced_hashes(&dbs, &mut report.databases)?;
+
+    // With no *store* database consulted, every file would look unreferenced.
+    // `.db` files that turned out not to be stores prove nothing, so this is
+    // checked after the collect rather than on the raw `.db` count.
+    if report.databases.is_empty() {
         return Err(crate::Error::NotFound(format!(
             "no store databases in {} — refusing to sweep",
             root.display()
         )));
     }
-
-    let referenced = collect_referenced_hashes(&dbs, &mut report.databases)?;

That also closes the third sub-finding: .filter_map(|e| e.ok()) dropped per-entry read_dir errors, so an unreadable store database would have gone unconsulted and its blobs deleted as orphans.

Dispatch the sweep before the store is opened

 pub fn run(args: DocsArgs) {
+    // Dispatched before the store is opened, unlike every other command:
+    // opening it creates and migrates a database in the very directory the
+    // sweep treats as its evidence, so a home with legacy blobs but no store
+    // left would gain a fresh empty one -- turning the sweep's "nothing to
+    // consult" refusal into a sweep with an empty reference set, which deletes
+    // every legacy blob.
+    if let DocsCmd::SweepLegacyBlobs { dry_run } = &args.cmd {
+        let root = flare_docs::DocsStore::default_db_path();
+        let root = root.parent().unwrap_or(std::path::Path::new("."));
+        match agentflare_store::maintenance::sweep_legacy_blobs(root, *dry_run) {
+            Ok(report) => println!("{}", serde_json::to_string_pretty(&report).unwrap()),
+            Err(e) => {
+                eprintln!("flare-docs: legacy blob sweep failed: {e}");
+                std::process::exit(1);
+            }
+        }
+        return;
+    }
+
     let store = match flare_docs::DocsStore::open_default() {

Coverage

Added the_sweep_refuses_when_no_database_present_is_a_store — a legacy blob directory whose only database has no store_blobs table must be refused with nothing deleted. The existing "unrelated database" test kept a real store.db alongside, so it never exercised the empty-reference-set path.